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ABSTRACT 


As  software  systems  increase  in  size  and  complexity,  so  does  the  need  to 
predict  and  control  scope,  schedule,  and  costs.  The  United  States  General 
Accountability  Office  has  acknowledged  weaknesses  in  the  software  acquisition 
process.  Industry  data  indicates  that  improving  the  software  development 
process  can  have  significant  effect  on  a  project  team’s  ability  to  generate 
products  within  planned  scope,  schedule,  and  cost  estimates.  This  thesis  focus 
is  on  software  maintenance,  one  phase  of  the  Army’s  acquisition  process,  to 
demonstrate  that  stronger  management  practices  are  needed  to  make  better 
predictions  and  assessments  in  those  areas.  Two  software  maintenance 
projects  were  evaluated  for  success  in  project  management  performance  against 
CMMI  practices.  This  research  results  in  a  set  of  recommendations  and 
predicted  benefits  are  provided  for  use  by  the  organization  as  input  to  the  next 
process  improvement  effort. 
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I. 


INTRODUCTION 


As  software  systems  increase  in  size  and  complexity,  the  need  to  predict 
and  control  costs  increases  correspondingly.  The  United  States  General 
Accountability  Office  has  acknowledged  weaknesses  in  the  software  acquisition 
process  in  their  March  2004  “Report  to  the  Committee  on  Armed  Services,  U.  S. 
Senate,  Defense  Acquisitions,  Stronger  Management  Practices  Are  Needed  to 
Improve  DOD’s  Software-Intensive  Weapon  Acquisitions”  which  indicates  that 
"...  For  the  most  part,  in  the  programs  reviewed,  DOD  garnered  poor  results 
from  its  software  acquisition  process  because  it  has  not  employed  consistent 
practices  in  these  areas  [setting  product  requirements,  maintaining  a  disciplined 
development  process,  and  using  metrics]  [6]”.  Effective  definition  and  use  of 
software  metrics  in  the  software  development  process  can  assist  project 
managers  in  making  decisions  and  managing  risks  that  affect  the  extent  of 
project  scope,  schedule,  and  cost.  Collecting  scope,  schedule,  and  cost  estimate 
data,  and  then  comparing  actual  results  to  that  data  over  time  can  help  to  answer 
questions  about  the  progress  of  a  project  and  improve  the  accuracy  of  estimates 
for  future  projects.  If  the  scope,  schedule,  and  budget  are  not  progressing  as 
good  as  planned,  software  teams  and  managers  can  use  the  information  to  make 
more  informed  mid-course  decisions  to  help  manage  project  risks.  Changes  in 
software  development  project  costs  can  translate  into  major  program  level 
savings  or  losses,  in  terms  of  scope,  schedule,  cost,  and  quality.  The  ability  to 
predict  the  cost  of  a  software  project  (or  the  portions  of  large  systems  that  are 
made  up  of  software)  has  a  large  affect  on  the  success  of  requesting  and 
planning  the  funding  needed  in  such  a  way  that  actual  costs  turn  out  to  be  as 
close  as  possible  to  expected  costs. 

Many  DOD  organizations  use  Capability  Maturity  Models  developed  by  the 
Software  Engineering  Institute  to  assess  the  ability  to  manage  development, 
acquisition,  and  maintenance  (or  “health”  of  a  project)  of  products  [7],  The  value 
of  these  models  has  been  recognized  in  typical  government  contract  language, 
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which  recommends  that  bidders  have  achieved  (or  have  a  plan  to  achieve)  a 
minimum  process  maturity  level  as  one  of  the  bidding  requirements.  Edward 
Aldridge,  Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics), 
issued  a  March  2003  “Memorandum  for  Secretaries  of  the  Military  Departments” 
with  reference  to  Section  804  of  the  Bob  Stump  National  Defense  Authorization 
Act  for  Fiscal  Year  2003,  which  requires  establishment  of  software  acquisition 
process  improvement  programs  for  defense  agencies  that  manage  major 
acquisition  programs  [5],  In  particular,  the  Army  has  recognized  that  software 
engineering  is  an  area  that  is  in  need  of  attention  [9], 

As  a  result  of  these  military  directions,  software  management 
organizations  are  encouraged  to  implement  software  development  process 
improvement  programs.  Aside  from  improving  the  organizations’  ability  to  deliver 
the  most  and  best  products,  within  allotted  time  and  budget,  the  primary 
motivation  is  to  demonstrate  capability  for  bidding  on  future  contracts  [10], 

This  thesis  focus  is  on  software  maintenance,  one  phase  in  the  Army’s 
acquisition  process.  The  purpose  of  the  study  is  to  demonstrate  that  stronger 
management  practices  are  needed.  Two  software  maintenance  projects 
(referred  to  as  “Case  1”  and  “Case  2”)  were  evaluated  for  success  in  these  areas 
against  CMMI  practices  [2],  To  keep  the  results  manageable  for  this  thesis,  only 
three  CMMI  Process  Areas  were  considered.  Results  of  the  evaluations  were 
examined  to  reveal  answers  to  the  following  questions: 

•  Was  the  organization  able  to  manage  technical  and  managerial 
requirements?  Were  inconsistencies  between  requirements  and 
project  plans/products  identified  and  managed? 

•  Was  the  organization  able  to  establish  and  maintain  project  baselines? 

•  Was  the  organization  able  to  provide  an  understanding  of  project 
progress  so  that  corrective  action  could  be  taken  to  make  changes  in 
project  management  baselines? 
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•  If  not,  what  process  improvements  would  need  to  be  implemented  to 
be  proficient  in  these  practices? 

•  What  benefits  might  the  organization  realize  as  a  result  of  this  process 
implementation? 

For  proprietary  reasons,  the  organization  and  particular  projects  studied 
will  remain  anonymous.  While  both  project  teams  were  able  to  deliver  the 
software  within  the  planned  scope,  schedule,  and  cost  constraints,  more  effective 
project  management  may  have  helped  them  to  finish  sooner,  or  to  take  on  more 
scope  within  the  same  limits.  The  information  collected  by  this  study  will  be 
available  for  immediate  use  by  the  project  team  as  they  are  preparing  for  the  next 
software  maintenance  project,  while  at  the  same  time  working  towards  achieving 
Level  3  goals  of  the  CMMI. 

Note  that  the  informal  examination  of  project  management  practices 
against  three  CMMI  Process  Areas,  demonstrated  on  two  similar  software 
maintenance  projects  conducted  for  this  study,  will  be  referred  to  as  an 
“evaluation.”  This  term  was  chosen  to  distinguish  the  activity  from  an  “audit” 
which  implies  that  a  corrective  action  loop  is  included.  The  goal  was  simply  to 
evaluate  the  projects,  compare  actual  practices  to  related  CMMI  practices,  and 
compare  them  to  each  other  and  to  the  lessons  learned  reported  by  the  project 
teams.  The  term  “evaluation”  is  used  to  distinguish  this  activity  from  a  process 
improvement  “assessment”  which  implies  the  involvement  of  independent 
licensed  professionals.  Instead,  an  informal  method  was  developed  and  used  to 
extract  information  strictly  for  internal  use.  However,  it  is  expected  that  a  formal 
SEI  CMMI  assessment  would  yield  the  same  results. 

This  thesis  documents  the  steps  taken  to  set  up  and  perform  the  process 
evaluations.  As  a  result,  the  author  compiled  a  list  of  software  development 
project  management  process  improvement  suggestions  and  potential  benefits, 
which  includes  successful  achievement  of  the  goals  in  each  of  the  three  CMMI 
Process  Areas  considered. 
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A.  ASSERTIONS 

The  SEI’s  process  improvement  model  (the  CMMI)  is  an  accepted  and 
good  method  to  determine  the  level  of  process  maturity  in  a  software 
organization.  Once  conducted,  evaluation  results  can  be  used  to  target  specific 
areas  for  improvement.  A  comparison  of  the  lessons  learned  reported  by  project 
team  can  be  used  as  a  sanity  check  against  evaluation  results.  If  the  project 
team  reported  problems  that  are  related  to  deficient  practices  then  evaluation 
results  may  be  more  believable  to  them,  which  may  help  to  give  recommended 
improvements  more  credibility.  Finally,  if  process  improvement  suggestions  are 
implemented,  benefits  are  likely  (i.e.,  project  managers  should  be  able  to  rely  on 
quantitative  measurement  as  a  tool  to  build  greater  accuracy  and  confidence  into 
planning  and  progress  reporting). 

B.  SCOPE 

This  study  is  limited  to  process  improvement  of  project  management 
activities  and  excludes  discussion  of  any  other  areas  of  software  development 
process  improvement  or  software  development  activities.  Two  separate,  but 
similar,  real-world  software  maintenance  projects  were  evaluated  for  the  purpose 
of  developing  thesis  conclusions.  The  evaluations  consist  of  a  comparison  of  the 
two  projects’  management  practices  to  those  defined  in  three  particular  CMMI 
Process  Areas,  which  were  selected  because  they  describe  a  standard  for 
activities  that  relate  to  planning  and  tracking  requirements,  schedule,  and  cost. 
The  three  Process  Areas  consist  of:  1)  Requirements  Management,  2)  Project 
Planning,  and  3)  Project  Monitoring  and  Control.  The  primary  intention  of  the 
research  is  to  produce  a  relative  picture  of  the  two  projects,  and  the  differences 
between  actual  practices  and  the  selected  CMMI  practices  to  provide  a  launch 
point  for  process  improvement  efforts  internal  to  the  organization.  Since 
improvement  is  a  continuous  process,  further  validation  and  verification  of  thesis 
conclusions  (predicted  benefits)  will  require  implementation  and  analysis  of  the 
improvements  suggested. 
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C.  THESIS  ORGANIZATION 

Following  the  INTRODUCTION,  BACKGROUND,  and  METHOD  chapters, 
a  chapter  on  the  study  of  each  of  the  three  relevant  CMMI  Process  Areas  is 
introduced  (REQUIREMENTS  MANAGEMENT  STUDY,  PROJECT  PLANNING 
STUDY,  and  PROJECT  MONITORING  AND  CONTROL  STUDY).  Each  of  those 
chapters  presents  the  CMMI  evaluation  results  for  both  projects  evaluated,  the 
post-mortem  lessons  learned  reported  by  the  project  team,  an  analysis  of  the 
data,  and  suggestions  for  improvement  in  each  area.  The  POTENTIAL 
BENEFITS  chapter  is  then  provided  as  a  basis  for  process  improvement 
motivation  for  the  organization,  and  perhaps  for  other  Army  software 
maintenance  organizations,  or  Army  organizations  that  depend  on  software 
maintenance  organizations  for  product  delivery.  It  outlines  the  predicted  return 
values  that  could  result  from  implementation  of  the  suggested  process 
improvements. 
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II. 


BACKGROUND 


A  summary  of  some  basic  concepts,  either  used  by  the  projects  studied, 
or  used  as  a  standard  to  compare  project  practices  to,  is  provided  in  this  chapter. 
Specifically,  a  summary  of  CMMI  Levels  1-3  project  management  goals  and 
practices,  and  project  management  process  requirements  in  terms  of  Unified 
Modeling  Language  notation  are  presented  to  set  the  stage  for  this  research. 
This  information  was  considered  as  the  basis  for  the  evaluations  and  analysis  to 
follow. 

A.  PROJECT  MANAGEMENT  IN  THE  CONTEXT  OF  THE  CMMI 

The  CMMI  groups  Process  Areas  for  the  purpose  of  discussing  concepts 
and  interactions.  Overall,  there  are  seven  Level  2  Process  Areas  (PAs)  and 
fourteen  Level  3  PAs,  each  with  a  set  of  goals  defined  for  achievement. 


Maturity  Levels 


Figure  1.  “CMMI  Model  Components  (From  [1])” 


There  are  three  CMMI  Process  Areas  that  address  activities  surrounding 
the  estimation,  collection,  tracking,  and  reporting  of  the  project  management  data 
mentioned  above.  These  Level  2  PAs  are  Requirements  Management,  Project 
Planning,  and  Project  Monitoring  and  Control.  PAs  are  divided  into  Specific  and 
Generic  Practices,  as  shown  in  Figure  1  [1],  Specific  Practices  are  model 
components  that  are  considered  important  guidance  for  achieving  “specific” 
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goals.  Generic  goals  and  practices  are  required  model  components  that  apply  to 
all  Process  Areas.  Each  project  was  evaluated  in  all  three  PAs,  in  both  specific 
and  generic  practices.  A  more  detailed  description  of  the  three  process  areas  is 
provided  in  the  following  subsections. 

1.  Basic  Project  Management  Process  Areas 

The  CMMI  states,  “the  purpose  of  Project  Planning  (PP)  is  to  establish 
and  maintain  plans  that  define  project  activities”  and  “the  purpose  of  Project 
Monitoring  and  Control  (PMC)  is  to  provide  an  understanding  of  the  project’s 
progress  so  that  appropriate  corrective  actions  can  be  taken  when  the  project’s 
performance  deviates  significantly  from  the  plan  [2].”  Together,  these  two 
Process  Areas  are  part  of  the  set  associated  with  fundamental  project 
management  activities  and  thus  were  selected  for  this  research.  CMMI  presents 
the  whole  group  of  project  management  PAs  as  shown  in  Figure  2  [4], 


Figure  2.  “Basic  Project  Management  Process 
Areas  (From  [4])” 
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2.  Engineering  Process  Areas 

The  CMMI  states,  “The  purpose  of  Requirements  Management  (REQM)  is 
to  manage  the  requirements  of  the  project’s  products  and  product  components, 
and  to  identify  inconsistencies  between  those  requirements  and  the  project’s 
plans  and  work  products  [4].”  In  the  case  of  software  maintenance  performed  by 
the  organization,  a  requirement  refers  to  the  “Statement  of  Work”  which  is 
approved  by  the  customer.  It  includes  project  requirements  (schedule  and  cost 
constraints)  and  technical  requirements  (a  statement  of  work).  This  Process 
Area  is  part  of  a  group  defined  by  the  CMMI  as  the  “Engineering  Process  Areas.” 
Figure  3  [4]  shows  that  requirements  management  is  an  activity  that  affects  the 
engineering  process  and  continues  with  a  loop  of  feedback  and  control 
throughout  the  project. 


Figure  3.  “Engineering  Process  Areas  (From  [4])” 
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B.  PROJECT  MANAGEMENT  IN  THE  CONTEXT  OF  UML 

Another  way  to  view  process  requirements  is  to  use  Unified  Modeling 
Language,  which  provides  a  visual  representation  of  a  system  (in  this  case,  a 
project  management  system)  from  the  users  (users  of  the  process)  perspective. 
Abstraction  can  be  used  to  define  concepts  and  relationships  between  system 
components  without  cumbersome  detail. 

1.  Project  Management  System  Functions 

At  the  most  general  level,  a  Project  Management  System  can  be  shown 
as  a  collection  of  elements  in  Figure  4.  The  box  shows  the  boundary  of  the 
system,  the  class  diagrams  show  the  subsystems  whose  functionalities 
(functional  requirements  of  the  process)  can  be  expressed  as  use  cases  in  the 

following  subsections.  _ 

Project  Management  System 


Figure  4  Project  Management  System 
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Further  detailed  notation  of  each  subsystem  includes  actors  that  represent 
points  of  interaction.  Connections  between  these  symbols  describe  the 
interaction  needed  to  accomplish  the  objective  of  the  use  case  (subsystem 
behavior).  The  following  subsections  describe  each  element  of  the  project 
management  system.  These  diagrams  provide  visual  representation  of  the 
benchmark  to  be  used  as  a  comparison  to  the  process  performance  of  the 
projects  studied.  Each  element  in  the  subsystem  use  case  diagrams  addresses 
the  requirements  and  interactions  that  represent  the  behaviors  described  in  the 
CMMI  practices  for  that  Process  Area,  now  with  the  participants  (or,  actors) 
included. 

2.  Requirements  Management  Element 

Requirements  management  occurs  both  external  and  internal  to  the 
organization.  At  this  level  of  abstraction,  these  aspects  are  not  distinguishable. 
What  can  be  seen  is  that  for  this  subsystem,  three  actors  are  involved  in  the  five 
practices  shown  in  Figure  5. 
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Figure  5.  Requirements  Management  Use  Case  Diagram 

3.  Project  Planning  Element 

Project  Planning  includes  activities  both  internal  and  external  to  the 
organization.  At  this  level  of  abstraction  these  aspects  are  not  distinguishable. 
What  can  be  seen  is  that  for  this  subsystem,  two  actors  are  involved  in  the  eight 
practices  shown  in  Figure  6. 
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Figure  6.  Project  Planning  Use  Case  Diagram 

4.  Project  Monitoring  and  Control  Element 

Project  Monitoring  and  Control  includes  activities  both  internal  and 
external  to  the  organization.  At  this  level  of  abstraction  these  aspects  are  not 
distinguishable.  What  can  be  seen  is  that  for  this  subsystem,  two  actors  are 
involved  in  the  eight  practices  shown  in  Figure  7. 
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Figure  7.  Project  Monitoring  and  Control  Use  Case  Diagram 
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III.  METHOD 


Two  projects  were  selected  for  this  study.  The  projects  had  similar  project 
teams,  requirements,  schedules,  artifacts,  and  staff.  A  checklist  was  developed 
for  each  of  the  three  CMMI  Process  Areas  chosen  for  the  study  (see  sample  in 
Appendix).  A  rating  system  to  categorize  the  level  of  performance  of  each 
practice  was  developed,  utilizing  one  of  three  possible  outcomes  for  each 
practice  evaluated:  “Not  Performed,”  “Not  Performed  Adequately,”  “Performed 
Adequately  (or  Better).”  It  is  understood  that  two  of  the  performance  categories 
are  subjective,  and  were  designed  to  reveal  the  amount  of  improvement  that 
would  be  necessary  in  certain  areas.  Results  from  both  Case  1  and  Case  2  were 
recorded  on  the  same  evaluation  checklist.  Information  included  in  the  evaluation 
results  was  provided  to  and  reviewed  by  the  project  team  members  who 
participated  in  the  evaluation. 

A.  PROCESS  EVALUATION 

1.  Entry  Criteria 

The  conditions  (work  products,  activities,  or  events)  that  must  have  taken 
place  before  the  activities  described  in  the  Procedure  can  be  executed  are  as 
follows: 

•  Selection  of  the  CMMI  as  the  model  as  a  benchmark  for  comparison 
purposes 

•  Development  of  checklists,  rating  scale,  approach,  and  goals  of 
process  evaluations 

•  Ensure  that  the  project  team  participants  understand  the  criteria  (CMMI 
practices)  for  comparison,  the  goals,  and  the  rating  system  to  be  used 

•  Obtain  permission  from  the  organization’s  senior  management  for 
team  members  to  participate  in  reviewing  and  discussing  evaluation 
results 
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2.  Input 

Information,  work  products  used  by  this  procedure,  or  actions  that  are 
needed  before  the  procedure  can  be  executed  are  as  follows: 

•  Previous  project  lessons  learned  (The  lessons  were  derived  from 
comments  provided  by  project  team  members  on  an  initial  anonymous 
survey  and  from  subsequent  feedback  sessions  held  with  the  project 
team  to  discuss  the  survey’s  findings.) 

•  Project  artifacts  such  as  project  plans,  project/senior  management 
review/customer  meeting  minutes,  configuration  and  quality  assurance 
records,  etc. 

•  CMMI  Process  Area  Evaluation  Checklists  (see  Appendix  for  example) 

3.  Exit  Criteria 

The  list  of  the  conditions  (completed  work  products  or  actions)  that  are 
needed  before  the  procedure  can  be  considered  complete  and  the  next  activity 
can  begin  is  as  follows: 

•  Collection  of  information  such  that  a  basis  for  a  rating  on  each  practice 
can  be  formed 

•  Consensus  by  participants,  or  resolution  of  disputes  by  the  SEPG 
Lead,  on  evaluation  results 

4.  Output 

The  list  of  the  benefits  and  work  products  produced  or  actions  that  will 
result  when  the  three  evaluations  are  complete  is  as  follows: 

•  Better  understanding  of  the  processes  used  by  the  organization 

•  Better  understanding  of  the  CMMI 

•  Evaluation  results  for  two  software  maintenance  projects  in  terms  of 
suggested  CMMI  practices  for  three  chosen  Process  Areas 

•  Better  understanding  of  the  organization’s  current  capability 

•  Better  understanding  of  the  gaps  between  the  organization’s  practices 
and  suggested  CMMI  practices 


16 


•  Better  understanding  of  how  the  organization  could  do  process  audits 

•  Process  evaluation  checklists  that  can  be  tailored  later  to  add  more 
organization  specific  processes,  and  then  used  as  process  audit 
checklists  for  internal  CMMI  assessments 

•  Outlined  procedures  were  generated  for  some  procedures  that  were 
not  documented  prior  to  the  evaluation 

•  Process  Improvement  artifacts  (evaluation  results  demonstrate 
whether  the  organization  has  improved,  maintained,  or  digressed  in 
certain  areas)  from  project  to  project 

•  Draft  procedure  for  conducting  process  audits 

•  A  sense  of  accomplishment,  as  a  team,  for  having  reached  consensus 
together.  Project  team  members  worked  together  with  SEPG 
members  (some  wore  both  hats). 

5.  Procedure 

The  following  list  summarizes  the  steps  taken  to  conduct  the  process 
evaluations: 

•  Evaluation  team  participants  were  briefed  on  the  approach,  goals,  and 
review  checklists  by  the  evaluator  (author  of  this  paper).  A  consensus 
on  format,  content,  and  rating  system  of  the  checklists  was  obtained. 
Checklists  were  revised  by  the  evaluator  as  recommended. 

•  The  process  evaluator  reviewed  project  artifacts,  and  when  necessary, 
conducted  interviews  with  project  members  to  get  input  on  processes 
and  artifacts  used. 

•  The  process  evaluator  drafted  evaluation  results  based  on  research 
and  initial  feedback. 

•  Draft  evaluation  results  were  released  to  the  evaluation  team  for 
review. 
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•  The  evaluator  and  evaluation  team  met  to  discuss  results.  The  rating 
and  justification  for  each  rating  were  reviewed  for  each  practice,  and 
each  project.  The  evaluator  noted  clarification  and  identification  of 
missing  artifacts  or  supporting  information. 

•  The  evaluator  finalized  evaluation  results  by  making  recommended 
changes. 

•  The  evaluator  released  the  results  to  the  evaluation  team  for  final 
review,  and  to  a  few  additional  project  team  members  who  may  have 
been  able  to  offer  additional  feedback.  Once  all  feedback  was 
incorporated,  the  results  were  considered  final. 

•  The  evaluator  and  evaluation  team  presented  the  evaluation  results  to 
the  project  team  together. 

B.  ANALYSIS 

The  next  three  chapters  present  discussion  from  a  variety  of  perspectives 
on  observed  project  management  ability.  First,  a  summary  of  both  Case  1  and 
Case  2  CMMI  Process  Area  evaluation  results  is  presented.  Analysis  of  each 
project  is  then  presented.  In  addition  to  presentation  of  a  brief  summary  of 
evaluation  results,  each  analysis  section  includes  review  of  post  mortem  lessons 
learned  (which  were  reported  by  project  team  members).  Next,  a  discussion  on 
the  comparison  of  Case  1  to  Case  2  evaluation  results  shows  whether  or  not  the 
organization  has  improved,  maintained,  or  digressed  in  process  capability  given 
the  goals  and  practices  examined.  A  comparison  of  the  lessons  learned  to  the 
evaluation  results  was  made  to  determine  if  there  is  a  correlation  between  the 
comments  and  the  gap  in  practices.  The  purpose  is  to  determine  if  there  are 
similarities  between  the  lessons  learned  and  the  evaluation  results. 

Rating  categories  can  be  viewed  as  a  measure  of  the  amount  of 
improvement  that  will  be  required  for  the  organization  to  reach  success  in  each 
practice  (i.e.,  “Not  Performed”  means  there  is  room  for  every  improvement,  “Not 
Performed  Adequately”  means  there  is  still  room  for  significant  improvement,  and 
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“Performed  Adequately”  means  that  there  is  still  room  for  some  improvement). 
Process  improvement  is  never  done! 
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IV.  REQUIREMENTS  MANAGEMENT  STUDY 


The  results  and  analysis  of  the  requirements  management  process  for  the 
two  case  studied  are  shown  in  the  following  subsections.  An  outline  of  the  goals 
and  associated  practices  defined  for  the  CMMI  Requirements  Management 
Process  Area,  a  score  for  each  project  studied  (Case  1  and  Case  2),  and 
justification  for  each  rating  is  first  provided.  A  general  analysis  of  the  evaluation 
results  for  each  case  follows  the  data  presentation.  The  analysis  includes 
presentation  and  comparison  to  the  post-mortem  lessons  learned  to  determine  if 
there  are  correlations  with  the  evaluation  results.  Note  that  the  lessons  are 
unique  to  the  individual  projects,  and  that  only  the  lessons  that  apply  to  the  area 
of  study  are  included.  Finally,  a  comparison  of  the  two  cases  is  made  and  a  list 
of  suggested  improvements  is  provided. 

A.  CASE  1  AND  CASE  2  EVALUATION  RESULTS 

CMMI  interpretation  notes,  the  score,  and  the  basis  for  the  rating  for  the 
Case  1  and  Case  2  projects  are  provided  below: 

•  Specific  Goal  (SG)  1  -  “Requirements  are  managed  and 

inconsistencies  with  project  plans  and  work  products  are  identified.” 
o  Specific  Practice  (SP)  1.1  -  “Develop  an  understanding  with  the 
requirements  providers  on  the  meaning  of  the  requirements”  was 
“Performed  Adequately”  for  both  Case  1  and  Case  2. 

1.  “Requirements”  was  interpreted  as  the  statement  of  work, 
budget,  and  delivery  schedule  constraints. 

2.  The  organization  demonstrated  that  they  worked  externally 
with  the  industry  partner  and  customer  to  get  more  familiar 
with  the  requirements  for  both  projects.  The  allocation  of 
requirements  to  the  organization  and  industry  partner  was 
then  negotiated,  and  an  agreement  was  made  with  the 
customer. 
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3.  Internally,  development  and  test  personnel  were  involved  in 
validating  software  trouble  reports  before  they  were  added  to 
the  scope,  thereby  demonstrating  their  understanding  of  the 
project  scope. 

o  SP1.2  -  “Obtain  Commitment  to  the  requirements  from  the  project 
participants”  was  “Performed  Adequately  for  Case  1.  The 
practice  was  “Not  Performed  Adequately  for  Case  2. 

1.  Note  that  this  practice  was  interpreted  as  referring  to  “initial” 
commitment  made  to  a  specific  set  of  requirements. 

2.  The  organization  demonstrated  that  an  external  initial 
commitment  was  made  as  a  result  of  negotiation  of  the 
project  requirements  with  the  industry  partner,  and  then 
approved  by  the  customer. 

3.  The  organization  demonstrated  internal  commitment  by 
involving  the  organization  project  management,  functional 
leads  (software  development,  test,  CM,  and  SGA)  in  the 
planning  activities,  including  peer  review  of  the  project 
plan(s).  A  baseline  is  created  for  this  organization  following 
successful  senior  management  review  and  approval. 

4.  The  Case  1  project  team  was  successful  at  completing  the 
above  process.  The  Case  2  project  team  did  not  get  the 
project  plan  through  the  peer  review  process  until  very  late  in 
the  program,  and  did  not  get  senior  management  approval 
until  after  the  work  was  done,  thus  received  the  lesser  rating. 
As  a  result,  during  the  program  the  test  and  development 
teams  were  well  aware  of  the  schedule.  However,  the  CM 
and  SOA  groups  were  not  fully  involved  in  these  planning 
activities  and  were  forced  to  accommodate  the  schedule 
after  the  fact. 
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o  SP1.3  -  “Manage  changes  to  the  requirements  as  they  evolve 
during  the  project”  was  “Not  Performed  Adequately  for  both  Case 
1  and  Case  2. 

1 .  This  practice  was  interpreted  as  referring  to  changes  made 
after  initial  baselines  were  established. 

2.  At  the  program  level,  as  requirements  changed  for  any 
reason,  the  baseline  statement  of  work  was  reviewed, 
updated,  and  approved. 

3.  As  a  result,  the  internal  schedule  and  scope  were  updated, 
informally  for  both  Cases.  Peer  reviews  to  make  sure  all 
functional  teams  were  well  aware  of  the  changes,  and  the 
senior  management  approval  process  were  neglected  for  the 
Case  1  baseline.  The  Case  2  baseline  was  never 
established  to  begin  with. 

o  SP1.4  -  “Maintain  bi-directional  traceability  among  the 

requirements  and  the  project  plans  and  the  work  products”  was 
“Not  Performed  Adequately”  for  both  Case  1  and  Case  2. 

1.  This  practice  was  interpreted  as  referring  to  both  the  initial 
set  of  requirements,  and  to  any  changes  made  to  the 
requirements. 

2.  Final  code  and  test  procedure  peer  review  packages  from 
both  projects  demonstrate  that  affected  requirements  were 
accurately  identified,  and  indeed,  if  requirements  changes 
were  necessary,  system  requirements  were  affected 
accordingly. 

3.  However,  for  both  projects,  traceability  of  the  requirements 
as  stated  in  the  internal  project  plans  were  not  maintained 
via  the  planned  peer  review  or  senior  management  approval 
process,  thus,  the  lesser  rating  was  assigned. 
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o  SP1.5  -  “Identify  inconsistencies  between  the  (project  plans  and 
work  products)  and  the  requirements”  was  “Performed 
Adequately  for  Case  1.  The  practice  was  “Not  Performed 
Adequately”  for  Case  2. 

1 .  Note  parenthesis  added  for  interpretation  to  the  practice 
definition. 

2.  As  requirements  changed,  the  internal  schedule  was 
revised,  albeit  informally.  As  some  requirements  were 
eliminated,  others  were  added.  Development  and  test  plans 
were  revised  accordingly.  Changes  were  maintained 
informally  by  the  software  development  lead.  However,  the 
changes  were  not  added  to  the  project  plans  to  undergo  the 
planned  peer  review  and  senior  management  approval 
process. 

3.  Both  projects  were  successful  at  micro-managing  the 
requirements  (work  and  services  were  performed  by 
development,  test,  CM  and  SQA  groups).  However,  the 
macro-management  (coordination  of  activities)  was  missing 
for  the  Case  2  project.  In  fact,  there  was  a  major  schedule 
misunderstanding  between  the  organization  and  the  industry 
partner.  This  could  have  been  due  to  the  lack  of  a  dedicated 
project  manager.  The  project  team  was  able  to  recover 
anyway. 

4.  Note  that  because  of  the  small  size  of  the  project  ream 
(around  20  members),  a  high-level  of  daily  face-to-face 
communication  took  place  to  help  maintain  smooth 
interfaces.  In  addition,  many  of  the  same  project  members 
from  the  Case  2  team  worked  on  the  Case  1  project,  and  on 
a  similar  project  prior  to  the  work  done  for  Case  1 .  The  team 
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was  able  to  rely  on  established  relationships  and  processes 
to  help  with  the  success  of  the  project. 

•  Generic  Goal  (GG)  2  -  “The  process  is  institutionalized  as  a  managed 
process.” 

o  Generic  Practice  (GP)  2.1  -  “Establish  and  maintain  an 

organizational  policy  for  planning  and  performing  the  Requirements 
Management  process”  was  “Performed  Adequately”  for  both  Case 
1  and  Case  2. 

1.  The  organization  does  have  a  formulated  and  documented 
Requirements  Management  policy  that  is  used  by  the  project 
teams.  The  plans  for  each  of  the  projects  provided  a 
statement  declaring  that  no  special  process  tailoring  was 
needed  to  execute  the  plan. 

o  GP2.2  -  “Establish  and  maintain  the  plan  for  performing  the 
Requirements  Management  process”  was  “Performed 
Adequately”  for  both  Case  1  and  Case  2. 

1.  Both  project  plans  (which  included  the  plans  for  the  project’s 
requirements  management  process)  were  developed, 
reviewed,  and  agreed  to  by  team  members. 

2.  The  project  team  decided  that  it  was  immaterial  that  the 
Case  2  project  plan  did  not  complete  the  peer  review  or 
senior  management  approval  process  successfully. 

o  GP2.3  -  “Provide  adequate  resources  for  performing  the 
Requirements  Management  process,  developing  the  work  products, 
and  providing  the  services  of  the  process”  was  “Performed 
Adequately”  for  both  Case  1  and  Case  2. 

1.  Resources  for  conducting  the  projects  requirements 
management  process  were  assigned  by  the  project  plans, 
o  GP2.4  -  “Assign  responsibility  and  authority  for  performing  the 
process,  developing  the  work  products,  and  providing  the  services 
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of  the  Requirements  Management  process”  was  “Performed 
Adequately”  for  both  Case  1  and  Case  2. 

1.  Each  project  plan  had  a  “roles  and  responsibility”  section, 
which  assigned  the  responsibilities  and  authority  to  all  of  the 
requirements  management  activities. 

o  GP2.5  -  “Train  the  people  performing  or  supporting  the 
Requirements  Management  process  as  needed”  was  “Performed 
Adequately”  for  both  Case  1  and  Case  2. 

1.  Training  for  the  project  team  consisted  of  previous 
experience,  individual  education,  and  on  the  job  training  for 
more  current  experiences.  The  project  team  had  the  skills 
necessary  to  execute  the  requirements  management 
activities. 

2.  For  Case  2,  the  project  team  did  receive  training  on  changes 
to  the  development  and  test  environment  alongside  the 
industry  partner. 

o  GP2.6  -  “Place  designated  work  products  of  the  Requirements 
Management  process  under  appropriate  levels  of  configuration” 
was  “Not  Performed  Adequately  for  both  Case  1  and  Case  2. 

1.  The  internal  project  plans  (including  scope,  and  schedule 
documents)  and  other  work  products  were  stored  in  a  shared 
computer  folder  for  easy  access.  Many  copies  have  not  yet 
been  placed  into  the  “Rational  ClearCase”  database,  which 
is  the  organizations  designated  CM  tool,  thus  the  lesser 
rating  was  assigned. 

o  GP  2.7  -  “Identify  and  involve  the  relevant  stakeholders  of  the 
Requirements  Management  process  as  planned”  was  “Performed 
Adequately”  for  both  Case  1  and  Case  2. 

1.  Relevant  stakeholders  (external  and  internal)  were  all 
identified  in  the  project  plans.  While  the  plans  were  not 
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formally  maintained  via  the  peer  review  and  senior 
management  approval  processes,  all  were  involved  in  the 
discussions  and  decisions  made  regarding  which 
requirements  would  be  added,  changed,  or  deleted.  It  would 
have  been  better  if  the  formal  processes  had  been  followed, 
as  it  could  have  prevented  a  schedule  misunderstanding 
with  the  industry  partner  that  occurred.  The  project  team 
was  able  to  recover  anyway. 

o  GP2.8  -  “Monitor  and  control  the  Requirements  Management 
process  against  the  plan  for  performing  the  process  and  take 
appropriate  corrective  action”  was  “Performed  Adequately  for 
Case  1 ,  and  was  “Not  Performed  Adequately  for  Case  2. 

1.  As  requirements  changed,  the  internal  schedule  was 
revised,  albeit  informally.  As  some  requirements  were 
eliminated,  others  were  added.  Development  and  test  plans 
were  revised  accordingly.  Changes  were  maintained 
informally  by  the  software  development  lead.  However,  the 
changes  were  not  added  to  the  project  plans  to  undergo  the 
planned  peer  review  and  senior  management  approval 
process. 

2.  Both  projects  were  successful  at  micro-managing  the 
requirements  (work  and  services  were  performed  by 
development,  test,  CM  and  SQA  groups).  However,  the 
macro-management  (coordination  of  activities)  was  missing 
for  the  Case  2  project.  In  fact,  there  was  a  major  schedule 
misunderstanding  between  the  organization  and  the  industry 
partner.  This  could  have  been  due  to  the  lack  of  a  dedicated 
project  manager.  The  project  team  was  able  to  recover 
anyway. 
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3.  Note  that  because  of  the  small  size  of  the  project  team 
(around  20  members),  a  high-level  of  daily  face-to-face 
communication  took  place  to  help  maintain  smooth 
interfaces.  In  addition,  many  of  the  same  project  members 
from  the  Case  2  team  worked  on  the  Case  1  project,  and  on 
a  similar  project  prior  to  the  work  done  for  Case  1 .  The  team 
was  able  to  rely  on  established  relationships  and  processes 
to  help  with  the  success  of  the  project, 
o  GP2.9  -  “Objectively  evaluate  adherence  of  the  Requirements 
Management  process  against  its  process  description,  standards, 
and  procedures,  and  address  noncompliance”  was  “Not  Performed 
Adequately”  for  both  Case  1  and  Case  2  projects. 

1.  The  overarching  process  for  SQA  discussed  independently 
evaluating  the  requirements  management  process.  This 
was  done  by  SQA  at  project  meetings  and  senior 
management  reviews.  However,  since  a  formal  audit  was 
never  conducted  the  lesser  rating  was  assigned  for  both 
projects.  The  project  team  indicated  that  a  formal  audit 
could  help  to  improve  the  process. 

o  GP2.10  -  “Review  the  activities,  status,  and  results  of  the 
Requirements  Management  process  with  higher  level  management 
and  resolve  issues”  was  “Performed  Adequately”  for  Case  1 ,  and 
was  “Not  Performed  Adequately  for  Case  2. 

1.  Requirements  development  and  changes  were  discussed  at 
project  and  senior  management  meetings.  While  both 
meetings  were  held  diligently  for  the  Case  1  project,  the 
Case  2  project  team  conducted  only  two  senior  management 
reviews  early  on  in  the  project.  In  addition,  the  occurrence  of 
Case  2  project  meetings  also  dropped  off  dramatically  which 
prevented  this  practice,  and  thus  the  lesser  rating  was 
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assigned.  This  could  have  been  due  to  the  lack  of  a 
dedicated  project  manager. 

•  Generic  Goal  (GG)  3  -  “The  process  is  institutionalized  a  defined 
process.” 

o  GP  3.1  -  “Establish  and  maintain  the  description  of  a  defined 
Requirements  Management  process”  was  “Performed 
Adequately”  for  both  Case  1  and  Case  2  projects. 

1 .  The  organization  level  requirements  management  procedure 
is  a  general  process  description  that  was  used  by  the  project 
teams,  and  has  not  changed.  Both  project  plans  provided  a 
declaration  statement  indicating  that  no  special  process 
tailoring  was  needed  to  execute  the  project  activities, 
o  GP  3.2  -  “Collect  work  products,  measures,  measurement  results, 
and  improvement  information  derived  from  planning  and  performing 
the  Requirements  Management  process  to  support  the  future  use 
and  improvement  of  the  organization’s  processes  and  process 
assets”  was  “Not  Performed  Adequately ”  for  both  Case  1  and 
Case  2  projects. 

1.  For  both  projects,  the  software  lead  maintained  a  list  of  the 
number  of  initial  requirements,  how  many  were  deemed  not 
applicable,  and  how  many  were  added.  In  addition,  data  on 
the  estimates  of  difficulty  and  time  to  resolve  problems  were 
made,  and  compared  to  actual  time  spent.  While  this  data 
was  recorded  it  was  not  compiled  for  analysis  with  process 
improvement  in  mind,  thus,  the  lesser  rating  was  assigned. 

B.  CASE  1  ANALYSIS 

1.  Evaluation  Results 

The  following  observations  were  made  based  on  process  evaluation 
ratings  given: 
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•  100%  of  the  practices  were  performed  to  some  degree  (all  were  rated 
either  “Not  Performed  Adequately’  ,  or  “Performed  Adequately  - 
none  were  rated  “Not  Performed)."  In  terms  of  the  CMMI  Model,  all 
goals  in  this  Process  Area  have  been  achieved  since  all  practices  are 
being  performed  to  some  extent. 

•  Overall,  12/17  or  71%  of  the  practices  were  rated  “Performed 
Adequately."  This  describes  the  success  of  most  practices  associated 
with  each  of  the  three  goals  in  this  Process  Area  indicating  that  the 
improvements  will  not  need  to  be  concentrated  on  one  particular  goal, 
rather,  a  few  practices  in  each  area  will  need  to  be  improved. 

2.  Lessons  Learned  Reported 

The  lessons  learned  that  affect  the  requirements  management  process  is 
described  below. 

a.  Project  Requirements  Were  Not  Well  Understood 

Some  confusion  resulted  from  the  team  not  having  a  good 
understanding  of  project  plans  (including  project  goals,  schedules,  constraints, 
documentation  required,  processes  to  be  used,  the  “big  picture”  of  how  the 
project  management  interacts  with  the  industry  partner  and  customer,  and  any 
organization  goals  or  initiatives  to  improve  project  performance).  In  addition,  the 
project  team  did  not  know  exactly  where  to  look  for  project  data  on  the  share 
drive  to  obtain  up-to-date  information  on  status. 
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C.  CASE  2  ANALYSIS 

1.  Evaluation  Results 

The  following  observations  were  made  based  on  process  evaluation 
ratings  given: 

•  100%  of  the  practices  were  performed  to  some  degree  (all  were  rated 
either  “Not  Performed  Adequately”,  or  “Performed  Adequately”  -  none 
were  rated  “Not  Performed).”  All  practices  are  being  performed  to 
some  extent. 

•  Overall,  8/17  or  47%  of  the  practices  were  rated  “Performed 
Adequately.”  This  describes  the  success  of  most  practices  associated 
with  each  of  the  three  goals  in  this  Process  Area  indicating  that  the 
improvements  will  not  need  to  be  concentrated  on  one  particular  goal, 
rather,  a  few  practices  in  each  area  will  need  to  be  improved. 

2.  Lessons  Learned  Reported 

a.  It  Was  Hard  to  Get  Next  SOW  Accepted  by  Industry  Partner 

The  SOW  for  the  organization  was  not  finalized  and  approved 

before  work  began.  This  resulted  in  the  industry  partner  starting/completing  work 
on  issues  that  the  organization  had  planned  to  work  on,  and  in  fact,  had  started 
working  on. 

b.  There  Was  Not  Enough  Coordination  of  System  Integration 
Lab  Activities  with  Developers  and  Testers 

There  was  a  lack  of  lab  resources  when  needed  (i.e.,  lab  was  going 
down  so  machines  were  not  available  for  testing).  Testing  had  to  be  done  at  the 
industry  partner’s  lab  at  the  mercy  of  their  schedule. 

D.  COMPARISON  OF  CASE  1  TO  CASE  2 

A  comparison  was  done  to  determine  the  similarities  and  differences 
between  Case  1  and  Case  2  evaluation  results. 

1.  Goals 

The  goals  associated  with  the  Requirements  Management  Process  Area 
are  as  follows: 
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•  Specific  Goal  (SG)  1  -  Requirements  are  managed  and  inconsistencies 
with  project  plans  and  work  products  are  identified. 

•  Generic  Goal  (GG)  2  -  The  process  is  institutionalized  as  a  managed 
process. 

•  Generic  Goal  (GG)  3  -  The  process  is  institutionalized  as  a  defined 
process. 

The  achievement  by  Case  1  and  Case  2  in  terms  of  goal  success  can  be 

described  by  a  comparison  of  the  number  of  practices  given  the  “Performed 

Adequately”  rating  to  the  total  number  of  practices  for  that  goal.  The  results  of 

this  analysis  follow: 

Case  1  -  SG  1  -  3/5  or  60% 

GG  2 -8/10  or  80% 

GG  3  -  1/2  or  50% 

Case  2-  SGI -1/5  or  20% 

GG  2 -6/10  or  60% 

GG  3  -  1/2  or  50% 

No  overall  goal  improvements  can  be  demonstrated  by  this  data,  only 
digression  for  SG  1  and  GG  2. 


2.  Practices 


Practice 

Case  1 

Case  2 

Not 

Performed 

Adequately 

Performed 

Adequatel 

y 

Not 

Performed 

Adequately 

Performed 

Adequately 

SP  1.1 

X 

X 

SP  1.2 

X 

X 

SP  1.3 

X 

X 

SP  1.4 

X 

X 

SP  1.5 

X 

X 

GP  2.1 

X 

X 

GP  2.2 

X 

X 

GP2.3 

X 

X 

GP  2.4 

X 

X 

GP  2.5 

X 

X 

GP  2.6 

X 

X 

GP  2.7 

X 

X 

GP  2.8 

X 

X 

GP  2.9 

X 

X 

GP  2.10 

X 

X 
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GP  3.1 

X 

X 

GP  3.2 

X 

X 

Table  1  Summary  of  Requirements  Management  Evaluation  Results 

The  results  do  not  demonstrate  improvement  from  Case  1  to  Case  2  in 
any  requirements  management  practice  evaluated.  They  do  show  digression  in 
the  following  practices  (highlighted  rows): 

•  SP1 .2  Obtain  Commitment  to  the  requirements  from  the  project 
participants 

•  SP1 .5  Identify  inconsistencies  between  the  project  plans  and  work 
products  and  the  requirements 

•  GP2.8  -  Monitor  and  control  the  Requirements  Management  process 
against  the  plan  for  performing  the  process  and  take  appropriate 
corrective  action 

•  GP2.10  -  Review  the  activities,  status,  and  results  of  the  Requirements 
Management  process  with  higher-level  management  and  resolve 
issues. 

In  addition,  results  indicate  that  for  both  projects,  the  following  practices 
were  “Not  Performed  Adequately 

•  SP1 .3  -  Manage  changes  to  the  requirements  as  they  evolve  during 
the  project 

•  SP1 .4  -  Maintain  bi-directional  traceability  among  the  requirements  and 
the  project  plans  and  the  work  products 

•  GP2.6  -  Place  designated  work  products  of  the  Requirements 
Management  process  under  appropriate  levels  of  configuration 

•  GP2.9  -  Objectively  evaluate  adherence  of  the  Requirements 
Management  process  against  its  process  description,  standards,  and 
procedures,  and  address  noncompliance 

GP  3.2  Collect  work  products,  measures,  measurement  results,  and 
improvement  information  derived  from  planning  and  performing  the 
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Requirements  Management  process  to  support  the  future  use  and  improvement 
of  the  organization’s  processes  and  process  assets. 

3.  Comparison  of  Evaluation  Results  to  Lessons  Learned 

Examination  of  the  Requirements  Management  process  evaluation  results 
shows  that  both  projects  had  some  practices  that  were  assigned  the  “Not 
Performed  Adequately”  rating.  Looking  closer  at  the  comments  reveal  that  both 
lacked  the  use  of  the  peer  review,  senior  management  approval,  and  formal  CM 
process  on  project  requirements,  which  resulted  in  a  lack  of  maintaining 
traceability  of  project  requirements  to  project  plans. 

It  makes  sense  that  if  project  requirement  information  is  not  up  to  date  or 
missing  in  plans,  and  not  accessible,  the  project  team  reported,  “project 
requirements  were  not  well  understood”  (the  lesson  learned  for  Case  1 ). 

The  Case  2  project  had  additional  practices  that  received  the  “Not 
Performed  Adequately”  rating.  Further  examination  of  detailed  comments  for 
those  evaluation  results  shows  that  the  basic  problems  included  the  fact  that  the 
project  team  did  not  get  the  project  plan  through  the  peer  review  process  until 
very  late  in  the  program,  and  did  not  get  senior  management  approval  until  after 
the  work  was  done.  In  addition,  macro-management  (coordination  of  activities) 
was  missing  for  the  Case  2  project.  In  fact,  there  was  a  major  schedule 
misunderstanding  between  the  organization  and  the  industry  partner.  In  addition, 
the  Case  2  project  team  conducted  only  two  senior  management  reviews  early 
on  in  the  project,  and  the  occurrence  of  project  meetings  dropped  off 
dramatically.  The  team  indicated  that  this  could  have  been  due  to  the  lack  of  a 
dedicated  project  manager  and  Senior  Management  Reviews  were  delayed  while 
waiting  for  plan  completion. 

One  of  the  lessons  learned  for  Case  2  involved  lack  of  coordination 
between  lab  activities  with  developers  and  testers.  The  same  comment  can  be 
made  (as  above),  that  is,  if  project  requirement  information  is  not  up  to  date  or 
missing  in  plans  it  was  not  accessible.  As  a  result,  the  lack  of  coordination 

between  the  project  team  functional  groups  is  not  surprising.  For  Case  2,  this 
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could  have  been  due  to  the  lack  of  a  dedicated  project  manager.  Note  that 
because  of  the  small  size  of  the  project  team  (around  20  members)  a  high-level 
of  daily  face-to-face  communication  took  place  to  help  maintain  smooth 
interfaces.  In  addition,  many  of  the  same  project  members  from  the  Case  2  team 
worked  on  the  Case  1  project  (and  on  a  similar  project  prior  to  the  work  done  for 
Case  1).  The  team  was  able  to  rely  on  established  relationships  and  processes 
to  help  with  the  success  of  the  project. 

The  evaluation  results  also  showed  that  while  “[process  improvement] 
data  [this  time  referring  to  technical  requirements]  was  recorded,  it  was  not 
compiled  for  analysis  with  process  improvement  in  mind,”  and  that  a  formal 
process  audit  [addressing  handling  of  project  and  technical  requirements]  had 
not  been  conducted. 

The  other  Case  2  lesson  learned  (It  was  hard  to  get  the  next  SOW 
accepted  by  Industry  Partner)  resulted  from  the  fact  that  the  SOW  for  the 
organization  was  not  finalized  and  approved  before  work  began.  This  resulted  in 
the  industry  partner  starting/completing  work  on  issues  that  the  organization  had 
planned  to  work  on,  and  in  fact,  had  started  working  on  (duplication  of  effort). 
While  this  lesson  involves  technical  requirements  instead  of  project 
requirements,  the  lesson  is  similar  in  that  it  is  a  logical  result  if  requirement 
information  is  not  up  to  date  or  missing  in  (internal,  and/or  external)  plans. 

It  is  interesting  to  note  that  one  characteristic  of  a  CMMI  Level  1 
organization  is  that  people  create  their  own  processes  to  do  the  work,  creating 
the  potential  for  duplication,  and  even  conflict  later  on  in  the  project.  Looking 
only  at  the  instance  of  duplication  (the  organization  and  industry  partner  working 
on  the  same  technical  requirements)  it  might  appear  that  this  is  a  Level  1 
organization.  However,  Level  2  organizations  share  lessons  learned  and  best 
practices  across  projects,  or  across  the  organization.  Requirements 
Management  is  a  Level  2  Process  Area.  While  the  organization  did  not  have 
outstanding  performance  on  all  practices,  it  was  demonstrated  that  something  is 

being  done  by  the  project  team  in  every  area.  The  transition  from  simply 
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managing  requirements  to  actually  managing  plans  takes  place  as  the 
organization  advances  from  Level  2  to  Level  3.  It  appears  that  the  source  of  risk 
lies  within  managing  the  plans  for  this  organization. 

This  information  can  be  used  to  identify  a  business  goal  on  which  to  base 
targeted  process  improvement,  such  as  “to  improve  the  project  requirements 
baseline  and  change  process.” 

E.  SUGGESTED  IMPROVEMENTS 

As  a  result  of  the  above  Case  1  and  Case  2  analysis,  the  following  list  of 
improvement  suggestions  was  compiled.  Some  suggestions  are  a  direct 
reflection  of  the  lessons  learned  reported,  some  are  specific  to  CMMI  practice 
requirements  that  need  to  be  fulfilled,  and  some  are  more  general  and  would 
contribute  indirectly  to  all  reported  deficiencies.  The  list  can  be  used  to  improve 
the  project  performance  by  targeting  process  improvement  in  these  areas. 

•  Make  sure  that  a  project  manager  is  assigned  and  dedicated  to 
initiating  and  following  through  on  requirements  management  activities 
for  the  project. 

•  Hold  a  project  kick-off  meeting  with  the  project  staff  to  brief  the  team 
on  the  project  plan(s).  Understanding  the  project  requirements  will 
help  to  alleviate  some  misunderstanding  within  the  project  team  and 
focus  on  successfully  achieving  them  in  a  more  efficient  manner. 

•  Make  sure  that  project  plans  (including  schedule  and  scope 
requirements  and  constraints)  are  reviewed  and  approved  by 
functional  leads  and  senior  management  early  on  in  the  project 
(complete  the  peer  review  and  management  approval  process  for 
managerial  and  technical  requirements). 

•  There  needs  to  be  more  coordination  between  the  organization’s  lab 
and  project  work.  A  work  plan  where  all  activities  are  scheduled  and 
coordinated  with  all  affected  parties  should  be  developed.  If 
communication  within  the  project  team  fails  and  there  isn’t  a  project 
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manager  to  keep  things  on  track,  senior  management  must  step  in  to 
avoid  negative  consequences. 

•  The  organization  needs  to  get  involved  with  the  industry  partner  earlier 
in  the  planning  stages.  In  fact,  the  organization  should  brief  the 
industry  partner  on  the  goals  of  the  partnership. 

•  Develop  questions  and  metrics  to  measure  accomplishments  in 
response  to  the  business  goal  to  “improve  the  project  requirements 
baseline  and  change  process.” 

•  Make  sure  project  plans  (including  schedule  and  scope  documents) 
are  updated  to  reflect  program  level  changes.  Use  “baselining” 
(accomplished  at  the  organization  by  completing  the  senior 
management  approval  process)  to  keep  a  record  of  current 
agreements  made  for  the  duration  of  the  project. 

•  Make  sure  that  project  and  senior  management  review  meetings  are 
held  as  planned,  on  a  regular  basis. 

•  Place  all  requirements  management  work  products  under  formal  CM 
control. 

•  Conduct  a  formal  audit  on  the  requirements  management  process  at 
least  once  during  project  execution. 

•  Update  policy,  process,  and  procedure  documentation  to  reflect  the 
additional  activities  recommended. 
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V. 


PROJECT  PLANNING  STUDY 


The  results  and  analysis  of  the  project  planning  process  for  the  two  case 
studied  are  shown  in  the  following  subsections.  An  outline  of  the  goals  and 
associated  practices  defined  for  the  CMMI  Project  Planning  Process  Area,  a 
score  for  each  project  studied  (Case  1  and  Case  2),  and  justification  for  each 
rating  is  first  provided.  A  general  analysis  of  the  evaluation  results  for  each  case 
follows  the  data  presentation.  The  analysis  includes  presentation  and 
comparison  to  the  post-mortem  lessons  learned  to  determine  if  there  are 
correlations  with  the  evaluation  results.  Note  that  the  lessons  are  unique  to  the 
individual  projects,  and  that  only  the  lessons  that  apply  to  the  area  of  study  are 
included.  Finally,  a  comparison  of  the  two  cases  is  made  and  a  list  of  suggested 
improvements  is  provided. 

A.  CASE  1  AND  CASE  2  EVALUATION  RESULTS 

The  goals  and  associated  practices  defined  for  the  CMMI  Project  Planning 
Process  Area,  and  individual  practice  ratings  assigned  by  the  project  team  are  as 
follows: 

•  Specific  Goal  (SG)  1  -  “Estimates  of  project  planning  parameters  are 
established  and  maintained.” 

o  Specific  Practice  (SP)  1.1  -  “Establish  a  top-level  work  breakdown 
structure  (WBS)  to  estimate  the  scope  of  the  Project”  was 
“Performed  Adequately”  for  both  Case  1  and  Case  2. 

1.  “Established”  was  interpreted  as  formulated,  documented, 
and  used,  and  not  necessarily  baselined. 

2.  The  initial  Work  Breakdown  Structure  for  both  projects  was 
identified  in  the  project  schedule. 

3.  Case  1  schedule  underwent  and  passed  the  peer  review  and 
senior  management  approval  process. 

4.  Case  2  schedule  did  not  undergo  the  peer  review  or  senior 
management  approval  process.  However,  the  project  team 
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indicated  that  the  project  team  was  well  aware  and  working 
to  the  proposed  WBS.  The  WBS  was  presented  at  the 
senior  management  review  meetings,  two  of  which  were 
held  early  on  in  the  project. 

o  SP  1.2  -  “Establish  and  maintain  estimates  of  the  attributes  of  the 
work  products  and  tasks”  was  “Performed  Adequately”  for  both 
Case  1  and  Case  2. 

1 .  “Attributes”  was  interpreted  as  “inputs  to  estimating  models, 
such  as  complexity,  or  skills/effort,  or  estimates  for  similar 
previous  experience  (values  that  affect  calculation  of  the  size 
of  the  project).” 

2.  For  both  Case  1  and  Case  2,  attributes  identified  were 
collected  and  maintained  by  the  functional  leads  (software 
development,  test,  CM,  and  QA)  on  an  informal  basis.  This 
information  was  shared  between  the  leads,  and  some  had 
been  combined  in  the  project  plan. 

o  SP1.3  -  “Define  the  project  life-cycle  phases  upon  which  to  scope 
the  planning  effort”  was  “Performed  Adequately  for  both  Case  1 
and  Case  2. 

1.  For  both  cases,  project  life  cycle  phases  were  described  in 
the  project  plans  as  task  categories  under  which  effort  would 
be  organized  for  recording  actual  time  spent.  Estimates 
were  separated,  and  actual  effort  recorded,  under  these 
categories. 

o  SP1.4  -  “Estimate  the  project  effort  and  cost  for  the  work  products 
and  tasks  based  on  estimation  rationale”  was  “Performed 
Adequately”  for  both  Case  1  and  Case  2. 

1 .  For  both  projects,  cost  and  effort  estimates  were  provided  in 
the  project  plans,  and  schedule  estimates  in  the  project 
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schedules.  As  noted  in  the  project  plans,  historical  data  was 
used  to  generate  the  estimates. 

2.  For  Case  2,  the  QA  Lead  documented  the  comparison  and 
analysis  of  Case  1  estimated  and  actual  effort  data  to 
generate  estimates  for  Case  2.  Other  functional  leads  called 
upon  previous  experience,  instead  of  examining  actual 
values  collected,  to  develop  their  estimates. 

•  Specific  Goal  (SG)  2  -  “A  project  plan  is  established  and  maintained  as 
the  basis  for  managing  the  project.” 

o  Specific  Practice  (SP)  2.1  -  “Establish  and  maintain  the  project’s 
budget  and  schedule”  was  “Not  Performed  Adequately  for  Case 
1  or  Case  2. 

1 .  For  both  projects,  cost  and  effort  estimates  were  provided  in 
the  project  plans,  schedule  estimates  in  the  project 
schedules. 

2.  For  Case  1 ,  estimates  underwent  the  peer  review  and  senior 
management  approval  process.  However,  estimates  were 
not  revised  when  there  was  a  significant  scope  increase 
early  in  the  project,  nor  for  the  remainder  of  the  project. 

3.  For  Case  2,  the  estimates  did  not  go  through  the  peer  review 
or  senior  management  approval  processes,  therefore,  were 
never  baselined  (nor  maintained). 

o  SP2.2  -  “Identify  and  analyze  project  risks”  was  “Performed 
Adequately”  for  Case  1  and  “Not  Performed  Adequately”  Case  2. 

1.  For  both  projects,  initial  risks  were  identified  in  the  project 
plans,  which  for  Case  1  underwent  the  peer  review  and 
senior  management  approval  process  (Case  2  risks  did  not 
go  through  these  processes). 

2.  For  both  projects  initial  risks  were  reviewed  and  updated 
regularly  at  project  and  senior  management  review  meetings 
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held  early  in  the  projects.  A  “risk  exposure”  was  calculated 
to  sum  up  the  risks  identified  for  each  project. 

3.  Subsequent  review  and  revision  of  this  data  took  place  at 
regular  project  and  senior  management  review  meetings. 
However,  this  activity  dropped  off  early  in  the  project  for 
Case  2  as  the  leadership  in  the  project  engineer  (manager) 
role  also  dropped  off. 

o  SP2.3  -  “Plan  for  the  management  of  project  data”  was 
“Performed  Adequately”  for  both  Case  1  and  Case  2. 

1 .  Plans  for  managing  project  data  were  documented  in  the 
project  plans  for  each  project. 

o  SP2.4  -  “Plan  for  the  necessary  resources  to  perform  the  project” 
was  “Performed  Adequately  or  both  Case  1  and  Case  2. 

1 .  Plans  for  resource  allocation  were  documented  in  the  project 
plans  for  each  project. 

o  SP2.5  -  “Plan  for  knowledge  and  skills  needed  to  perform  the 
project”  was  “Performed  Adequately”  for  both  Case  1  and  Case  2. 

1.  Plans  for  training  needed  to  accomplish  the  work  were 
documented  in  the  project  plans  for  each  project. 

o  SP2.6  -  “Plan  the  involvement  of  identified  stakeholders”  was 
“Performed  Adequately”  for  both  Case  1  and  Case  2. 

1 .  Identification  of  (and  plans  for  interaction  with)  major 
stakeholders  was  documented  in  the  project  plans  for  each 
project. 

o  SP2.7  -  “Establish  and  maintain  the  overall  project  plan  content” 
was  “Not  Performed  Adequately  for  both  Case  1  and  Case  2. 

1.  For  Case  1,  the  project  plan  was  developed,  and  passed  the 
peer  review  and  senior  management  approval  processes. 
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2.  For  Case  2,  the  plan  was  developed.  However,  it  did  not  go 
through  the  same  review  and  approval  processes,  so  an 
initial  baseline  was  never  established. 

3.  While  for  both  Case  1  and  Case  2,  project  plans  were 
established,  they  were  not  maintained  for  either  project  (to 
reflect  changes  in  scope,  schedule,  or  effort  estimates). 

•  Specific  Goal  (SG)  3  -  “Commitments  to  the  project  plan  are 

established  and  maintained.” 

o  Specific  Practice  (SP)  3.1  -  “Review  all  plans  that  affect  the  project 
to  understand  project  commitments”  was  “Performed  Adequately 
for  both  Case  1  and  Case  2. 

1.  For  Case  1,  the  project  plan  was  developed,  and  passed  the 
peer  review  and  senior  management  approval  processes. 

2.  For  Case  2,  the  plan  was  developed,  and  underwent  the 
peer  review  process. 

o  SP3.2  -  “Reconcile  the  project  plan  to  reflect  available  and 
estimated  resources”  was  “Performed  Adequately  for  both  Case 
1  and  Case  2. 

1.  For  Case  1,  the  project  plan  was  developed,  and  passed  the 
peer  review  and  senior  management  approval  processes. 

2.  For  Case  2,  the  plan  was  developed,  and  underwent  the 
peer  review  process. 

o  SP3.3  -  “Obtain  [initial]  commitment  from  relevant  stakeholders 
responsible  for  performing  and  supporting  plan  execution”  was 
“Performed  Adequately  for  Case  1  and  “Not  Performed 
Adequately  for  Case  2. 

1.  For  Case  1,  the  project  plan  was  developed,  and  passed  the 
peer  review  and  senior  management  approval  processes. 

2.  For  Case  2,  the  plan  was  developed,  and  underwent  the 
peer  review  process,  but  not  until  well  into  the  performance 
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period.  It  was  not  approved  by  senior  management  until 
after  the  project  work  was  done. 

3.  The  team  lead  (senior  management)  was  involved  in  several 
informal  reviews  and  the  peer  review.  However,  the  plan 
was  not  approved  until  after  the  project  work  was  done. 

•  Generic  Goal  (GG)  2  -  “The  process  is  institutionalized  as  a  managed 

process.” 

o  Generic  Practice  (GP)  2.1  -  “Establish  and  maintain  an 

organizational  policy  for  planning  and  performing  the  Project 
Planning  process”  was  “Performed  Adequately  for  both  Case  1 
and  Case  2. 

1.  The  organization  has  a  project  planning  policy  albeit  very 
general.  The  process  did  not  change  from  the  time  the  Case 
1  project  was  executed  to  the  time  the  Case  2  project  was 
executed. 

o  GP2.2  -  “Establish  and  maintain  the  plan  for  performing  the  Project 
Planning  process”  was  “Performed  Adequately  for  Case  1  and 
“Not  Performed  Adequately  for  Case  2. 

1 .  The  organization  project  plan  procedures  describe  how  to 
create  a  project  plan. 

2.  Schedules  for  both  Case  1  and  Case  2  projects  have 
development  milestones  for  the  plans,  under  a  main  “Project 
Planning”  category. 

3.  For  Case  1 ,  the  project  plan  was  developed,  and  passed  the 
peer  review  and  senior  management  approval  processes. 

4.  For  Case  2,  the  plan  was  developed  but  did  not  go  through 
the  same  review  and  approval  processes. 

o  GP2.3  -  “Provide  adequate  resources  for  performing  the  Project 
Planning  process,  developing  the  work  products,  and  providing  the 
services  of  the  process”  was  “Performed  Adequately  for  Case  1 
and  “Not  Performed  Adequately  for  Case  2. 
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1 .  Resources  were  provided  for  both  projects  to  do  planning,  as 
evidenced  by  the  project  plans  developed. 

2.  However,  for  Case  2,  primary  responsibility  was  passed  from 
one  to  another  person  more  than  once.  When  action  wasn’t 
taken,  no  “quick”  attempt  was  made  by  management  to 
reassign  this  responsibility. 

o  GP2.4  -  “Assign  responsibility  and  authority  for  performing  the 
process,  developing  the  work  products,  and  providing  the  services 
of  the  Project  Planning  process”  was  “Performed  Adequately”  for 
Case  1  and  “Not  Performed  Adequately  for  Case  2. 

1 .  Resources  were  provided  for  both  projects  to  do  planning,  as 
evidenced  by  the  project  plans  developed. 

2.  However,  for  Case  2,  primary  responsibility  was  passed  from 
one  to  another  person  more  than  once.  When  action  wasn’t 
taken,  no  “quick”  attempt  was  made  by  management  to 
reassign  this  responsibility. 

o  GP2.5  -  “Train  the  people  performing  or  supporting  the  Project 
Planning  process  as  needed”  was  “Performed  Adequately  for 
both  Case  1  and  Case  2. 

1 .  For  both  projects,  training  consisted  of  previous  experience 
of  the  members  of  a  small  management  team;  in  fact,  the 
same  team  was  used  for  both  projects. 

2.  In  addition,  on  the  job  training  was  provided  to  some  team 
members  by  functional  leads,  and  some  had  a  college 
course  in  project  management,  which  included  planning 
aspects. 

o  GP2.6  -  “Place  designated  work  products  of  the  Project  Planning 
process  under  appropriate  levels  of  configuration”  was  “Not 
Performed  Adequately  for  Case  1  or  Case  2. 
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1.  All  project  plans,  schedules,  and  other  project  planning 
artifacts  were  stored  in  the  team  shared  folder.  However, 
the  documents  were  not  put  into  ClearCase,  the  designated 
CM  tool. 

2.  The  Case  2  project  plan  was  put  into  ClearCase,  but  not  until 
after  the  project  was  over. 

o  GP  2.7  -  “Identify  and  involve  the  relevant  stakeholders  of  the 
Project  Planning  process  as  planned”  was  “Performed 
Adequately”  for  both  Case  1  and  Case  2. 

1.  Relevant  stakeholders  were  all  part  of  the  team  that 
developed  and  reviewed  the  project  plans  for  both  projects. 

2.  These  activities  were  monitored  by  the  QA  group,  during 
project  and  senior  management  review  meetings. 

o  GP2.8  -  “Monitor  and  control  the  Project  Planning  process  against 
the  plan  for  performing  the  process  and  take  appropriate  corrective 
action”  was  “Performed  Adequately  for  both  Case  1  and  Case  2. 

1.  Project  plans  for  both  Case  1  and  Case  2  underwent  the 
peer  review  process.  However,  corrective  action  was  not 
taken  to  address  getting  plan  changes  incorporated. 

2.  These  activities  were  monitored  by  the  QA  group,  during 
project  and  senior  management  review  meetings. 

o  GP2.9  -  “Objectively  evaluate  adherence  of  the  Project  Planning 
process  against  its  process  description,  standards,  and  procedures, 
and  address  noncompliance”  was  “Performed  Adequately  for 
both  Case  1  and  Case  2. 

1.  Planning  activities  were  monitored  (albeit  informally)  by  the 
QA  group,  during  project  and  senior  management  review 
meetings. 

o  GP2.10  -  “Review  the  activities,  status,  and  results  of  the  Project 
Planning  process  with  higher-level  management  and  resolve 
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issues”  was  “Performed  Adequately”  for  Case  1  and  “Not 
Performed  Adequately”  for  Case  2. 

1 .  Senior  management  review  meetings  were  conducted  for  the 
Case  1  project  on  a  regular  basis  throughout  the  entire 
project. 

2.  Only  2  senior  management  review  meetings  were  conducted 
early  on  for  the  Case  2  project. 

Generic  Goal  (GG)  3  -  “The  process  is  institutionalized  a  defined 

process.” 

o  GP  3.1  -  “Establish  and  maintain  the  description  of  a  defined 
Project  Planning  process”  was  “Performed  Adequately  for  both 
Case  1  and  Case  2. 

1.  Four  organization  level  procedures  were  identified;  the 
“Software  Schedule  Procedure”,  the  “Senior  Management 
Review  Procedure”,  the  “Software  Development  Plan 
Procedure”,  and  the  “Software  Project  Planning  and 
Estimating  Procedure.” 

o  GP  3.2  -  “Collect  work  products,  measures,  measurement  results, 
and  improvement  information  derived  from  planning  and  performing 
the  Project  Planning  process  to  support  the  future  use  and 
improvement  of  the  organization’s  processes  and  process  assets” 
was  “Not  Performed  Adequately”  for  Case  1  or  Case  2. 

1.  Metrics  on  the  planning  process  (such  as  the  number  of 
pages  for  the  project  plan,  length  of  time  to  prepare  the  plan, 
and  number  of  peer  reviews  held  for  previous  projects)  was 
available.  However,  the  data  was  not  used  to  generate 
estimates  for  either  Case  1  or  Case  2. 

2.  Lessons  learned  from  previous  projects  and  for  both  of  these 
projects  were  identified,  collected,  and  analyzed  to  assist  in 
planning  the  next  project. 


3.  All  of  these  activities  were  done  on  an  informal  basis.  Data 
was  not  complied  and  analyzed  with  process  improvement  in 
mind. 

B.  CASE  1  ANALYSIS 

1.  Evaluation  Results 

The  following  observations  were  made  based  on  ratings  assigned: 

•  100%  practices  were  performed  to  some  degree  (all  were  rated  either 
“Not  Performed  Adequately”,  or  “Performed  Adequately  -  none 
were  rated  “Not  Performed)."  All  practices  are  being  performed  to 
some  extent. 

•  Overall  22/26  or  85%  practices  were  rated  “Performed  Adequately . 
This  describes  the  success  of  most  practices  associated  with  each  of 
the  three  goals  in  this  Process  Area  indicating  that  the  improvements 
will  not  need  to  be  concentrated  on  one  particular  goal. 

2.  Lessons  Learned  Reported 

a.  The  Project  Lacked  a  Single  Unified  Software  Project 
Management  Plan 

The  lack  of  a  single  coordinated  plan  resulted  in  individual 
functional  plans  whose  schedules  were  not  necessarily  in  synch.  This  led  to 
significant  delays  in  getting  senior  management  approval. 

b.  The  Lab  Facility  Had  Much  Unplanned  Lab  Down  Time 

Organization  lab  facility  schedules  and  resources  directly  affected 
the  development  and  test  efforts,  in  fact,  forced  the  need  to  utilize  resources  at 
the  industry  partner  facility  at  their  convenience  (with  some  inconvenience  to  the 
developers  and  testers).  As  a  result  there  were  significant  delays  in  development 
and  testing  activities  that  required  lab  use. 
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C.  CASE  2  ANALYSIS 

1.  Evaluation  Results 

•  100%  practices  were  performed  to  some  degree  (all  were  rated  either 
“Not  Performed  Adequately”,  or  “Performed  Adequately  -  none 
were  rated  “Not  Performed  ).  All  practices  were  performed  to  some 
extent. 

•  Overall,  16/26  or  62%  practices  were  rated  “Performed  Adequately”. 
This  describes  the  success  of  most  practices  associated  with  each  of 
the  three  goals  in  this  Process  Area  indicating  that  the  improvements 
will  not  need  to  be  concentrated  on  one  particular  goal. 

2.  Lessons  Learned  Reported 

a.  The  Project  Plan  Took  too  long  to  Produce  and  Review 

The  project  plan  was  not  completed,  peer  reviewed,  and  approved 
by  senior  management  until  well  after  the  work  was  done.  New  persons,  the 
industry  partner,  and  even  existing  team  members  did  not  know  about  and 
understand  all  parts  of  the  plan  so  there  was  confusion  about  several  aspects, 
especially  how  one  group  handed  off  their  work  to  the  next  group  (development, 
test,  SQA,  CM). 

b.  Effort  Estimates  Were  Not  Revised  to  Reflect  Increased 
Scope 

Early  on  in  the  project  there  was  a  significant  increase  in  the  scope. 
Effort  estimates  were  not  revised,  peer  reviewed,  and  approved  by  senior 
management.  As  a  result,  after  that  point  it  was  impossible  to  compare  actual  to 
estimated  data  (it  did  not  make  sense  anymore. 

c.  Effort  Data  Was  Not  Organized  Consistently 

There  does  not  appear  to  be  a  consistent  way  to  organize  the  data 
(i.e.,  to  record  time  spent  on  multiple  projects,  or  for  multiple  tasks  on  the  same 
project).  SQA  records  ALL  data  for  100%  hours  spent  in  one  worksheet  for  each 
group  member.  This  enables  overhead/direct  effort  and  project-to-project 
comparisons.  There  appears  to  be  a  separate  worksheet  for  some  activities. 

Yet,  the  data  it  contains  has  not  been  considered  in  audits.  In  some  cases  there 
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is  no  way  to  determine  which  project  the  data  is  recorded  for,  and  the  existing 
data  does  not  appear  to  be  complete. 

D.  COMPARISON  OF  CASE  1  TO  CASE  2 

1.  Goals 

•  Specific  Goal  (SG)  1  -  Estimates  of  project  planning  parameters  are 
established  and  maintained. 

•  Specific  Goal  (SG)  2  -  A  project  plan  is  established  and  maintained  as 
the  basis  for  managing  the  project. 

•  Specific  Goal  (SG)  3  -  Commitments  to  the  project  plan  are  established 
and  maintained. 

•  Generic  Goal  (GG)  2  -  The  process  is  institutionalized  as  a  managed 
process. 

•  Generic  Goal  (GG)  3  -  The  process  is  institutionalized  a  defined 
process. 

Case  1  -  SG  1  -  4/4  or  100% 

SG  2  -  5/7  or  71  % 

SG  3  -  3/3  or  100% 

GG  2-8/10  or  80% 

GG  3  -  y2  or  50% 

Case  2-  SG  1  -4/4  or  100% 

SG  2  -  4/7  or  57% 

SG  3  -  2/3  or  67% 

GG  2-4/10  or  40% 

GG  3-  1/2  or  50% 

Unfortunately,  no  overall  goal  improvements  can  be  demonstrated  by  this 
data,  only  digression  for  SG  3  and  GG  2. 

2.  Practices 


Practice 

Case  1 

Case  2 

Not 

Performed 

Adequately 

Performed 

Adequatel 

y 

Not 

Performed 

Adequately 

Performed 

Adequately 

SP  1.1 

X 

X 

SP  1.2 

X 

X 

SP  1.3 

X 

X 

SP  1.4 

X 

X 

SP  2.1 

X 

X 
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SP  2.2 

X 

X 

SP  2.3 

X 

X 

SP  2.4 

X 

X 

SP  2.5 

X 

X 

SP  2.6 

X 

X 

SP  2.7 

X 

X 

SP  3.1 

X 

X 

SP  3.2 

X 

X 

SP  3.3 

X 

X 

GP  2.1 

X 

X 

GP  2.2 

X 

X 

GP  2.3 

X 

X 

GP  2.4 

X 

X 

GP  2.5 

X 

X 

GP  2.6 

X 

X 

GP  2.7 

X 

X 

GP  2.8 

X 

X 

GP  2.9 

X 

X 

GP  2.10 

X 

X 

GP  3.1 

X 

X 

GP  3.2 

X 

X 

Table  2  Summary  of  Project  Planning  Evaluation  Results 


The  results  do  not  demonstrate  improvement  from  Case  1  to  Case  2  in 
any  project  planning  practice  evaluated.  They  do  show  digression  in  the 
following  practices  (highlighted  rows); 

•  SP  2.2  -  Identify  and  analyze  project  risks. 

•  SP3.3  -  Obtain  commitment  from  relevant  stakeholders  responsible  for 
performing  and  supporting  plan  execution. 

•  GP2.2  -  Establish  and  maintain  the  plan  for  performing  the  Project 
Planning  process. 

•  GP2.3  -  Provide  adequate  resources  for  performing  the  Project 
Planning  process,  developing  the  work  products,  and  providing  the 
services  of  the  process. 
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•  GP2.4  -  Assign  responsibility  and  authority  for  performing  the  process, 
developing  the  work  products,  and  providing  the  services  of  the  Project 
Planning  process. 

•  GP2.10  -  Review  the  activities,  status,  and  results  of  the  Requirements 
Management  process  with  higher-level  management  and  resolve 
issues. 

In  addition,  results  indicate  that  for  both  projects,  the  following  practices 
were  “Not  Performed  Adequately:’’ 

•  SP  2.1  -  Establish  and  maintain  the  project’s  budget  and  schedule. 

•  SP2.7  -  Establish  and  maintain  the  overall  project  plan  content. 

•  GP2.6  -  Place  designated  work  products  of  the  Project  Planning 
process  under  appropriate  levels  of  configuration. 

•  GP  3.2  -  Collect  work  products,  measures,  measurement  results,  and 
improvement  information  derived  from  planning  and  performing  the 
Project  Planning  process  to  support  the  future  use  and  improvement  of 
the  organization’s  processes  and  process  assets. 

3.  Comparison  of  Evaluation  Results  to  Lessons  Learned 

Examination  of  the  comments  from  the  evaluation  results  for  which  both 
projects  had  “Not  Performed  Adequately”  rating  show  that  while  for  both  Case  1 
and  Case  2,  project  plans  were  established,  they  were  not  formally  maintained  or 
controlled  for  either  project  (to  reflect  changes  in  scope,  schedule,  or  effort 
estimates).  Some  metrics  were  collected,  some  artifacts  were  stored,  and  some 
lessons  learned  were  collected.  However,  those  activities  were  also  conducted 
informally,  without  process  improvement  necessarily  in  mind. 

The  Case  1  project  team  noted  negative  impacts  to  the  project  due  the 
lack  of  a  single  unified  software  project  management  plan,  and  the  fact  that  the 
lab  facility  had  much  unplanned  down  time  (lessons  learned).  As  a  result,  there 
were  schedule  and  resource  misunderstandings,  including  delays  in  development 
and  testing.  These  results  are  a  logical  conclusion  (could  have  been  predicted) 
given  the  process  evaluation  results. 
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The  Case  2  project  had  additional  practices  that  received  the  “Not 
Performed  Adequately”  rating  (noted  above).  The  Case  2  project  digressed 
quickly  by  never  establishing  a  requirements  baseline  (which  for  this  organization 
translates  into  following  the  management  approval  process),  by  not  following 
through  on  project  and  senior  management  review  meetings,  and  more  important 
by  not  having  a  dedicated  project  engineer  (manager).  The  project  team  and 
senior  management  knew  what  to  do  (they  had  been  done  well  for  the  Case  1 
project).  However,  they  lacked  the  discipline  required  to  follow  through  with 
previously  followed  processes.  Without  conducting  a  formal  project  planning 
process  audit,  the  project  management  activities  degraded  slowly,  without  much 
notice.  After  all,  the  team  worked  to  the  draft  project  plans  and  products  were 
getting  generated  in  spite  of  the  circumstances.  However,  when  project 
management  duties  fell  by  the  wayside  after  obvious  need  several  team 
members  became  frustrated 

The  Case  2  project  team  noted  that  the  Project  Plan  took  too  long  to 
produce  and  review,  effort  estimates  were  not  revised  to  reflect  increased  scope, 
and  effort  data  was  not  organized  consistently.  As  a  result,  there  were 
misunderstandings,  miscommunications,  and  a  general  lack  of  commitment  and 
interest  in  completing  the  project  plan,  and  recording  effort  data.  The  decisions 
(not)  made  put  the  team  in  the  position  whereby  it  became  necessary  to  rely  on 
the  small  size  and  experience,  instead  of  disciplined  project  management 
activities.  The  lack  of  team  and  senior  management  interaction  caused  the  team 
to  drift  apart  in  the  way  effort  data  was  organized  and  recorded. 

Again,  it  is  interesting  to  note  that  one  characteristic  of  a  CMMI  Level  1 
organization  is  that  people  create  their  own  processes  to  do  the  work,  creating 
the  potential  for  duplication,  and  even  conflict  later  on  in  the  project.  This  is 
exactly  what  happened  with  the  project  plan  and  the  effort  data  records. 

While  the  organization  did  not  have  outstanding  performance  on  all 
practices,  it  was  demonstrated  that  something  was  being  done  by  the  project 

team  in  every  process  area.  The  transition  from  simply  managing  requirements 
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to  actually  managing  plans  takes  place  as  the  organization  advances  from  Level 
2  to  Level  3.  It  appears  that  the  source  of  risk  lies  within  managing  the  plans  for 
this  organization. 


This  information  can  be  used  to  identify  a  business  goal  on  which  to  base 
targeted  process  improvement,  such  as  “to  improve  the  project  plan  baseline  and 
change  process.” 

E.  SUGGESTED  IMPROVEMENTS 

As  a  result  of  the  above  Case  1  and  Case  2  analysis,  the  following  list  of 
improvement  suggestions  was  compiled.  Some  suggestions  are  a  direct 
reflection  of  the  lessons  learned  reported,  some  are  specific  to  CMMI  practice 
requirements  that  need  to  be  fulfilled,  and  some  are  more  general  and  would 
contribute  indirectly  to  all  reported  deficiencies.  The  list  can  be  used  to  improve 
the  project  performance  by  targeting  process  improvement  in  these  areas. 

•  Make  sure  a  dedicated  resource  is  assigned  to  perform  the  project 
engineer  (manager)  role  (to  follow  up  on  project  plan  peer  review  and 
approval  process,  project  and  senior  management  review  meetings, 
the  change  control  process,  etc.).  If  a  dedicated  resource  is 
unavailable,  duties  should  be  reassigned  aross  available  resources  (in 
this  case,  Functional  Leads). 

•  A  single  Software  Project  Management  Plan  (SPMP)  should  be 
developed  that  coordinates  all  of  the  functional  area  plans  for  the 
project  -  Software  Development  Plan  (SDP),  Risk  Management  Plan 
(RMP),  Software  Test  Plan  (STP),  Software  Configuration 
Management  Plan  (SPMP),  Software  Quality  Assurance  Plan  (SQAP), 
and  Laboratory  Management  Plan  (LMP). 

o  The  schedule  and  resources  available  for  the  organization  lab 
facilities  should  be  included  in  planning  and  estimating  activities. 
The  acquisition  of  laboratory  equipment  and  any  related  down  time 
for  installation  should  be  included  in  overall  project  planning.  This 
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would  ensure  that  lower  priority  items  would  be  managed  at  the 
same  time  immediate  problems  are  being  resolved  quickly.  This 
would  prevent  significant  delays  in  development  and  testing 
activities  that  require  lab  use.  The  lab  played  a  huge  part  in  the 
ability  to  follow  the  planned  processes,  for  both  cases.  This  implies 
that  management  needs  to  improve  the  lab  situation  (i.e.,  provide 
more  staffing/resources/oversight). 

o  A  new  focused  process  for  producing  the  plan  is  needed.  The  plan 
can  be  stripped  down  to  its  essential  elements  to  further  reduce  the 
time  it  takes  to  produce  and  review.  Finish  the  plan  well  before 
delivering  software.  Define  specific  authority  and  criteria  to  catch  a 
significant  slip  in  plan  completion. 

•  Make  sure  project  plans  (including  schedule  and  scope  documents) 
are  updated  to  reflect  program  level  changes.  Use  “baselining” 
(accomplished  at  the  organization  by  completing  the  senior 
management  approval  process)  to  keep  a  record  of  current 
agreements  made  for  the  duration  of  the  project. 

•  Updates  to  effort  estimates  must  be  documented  and  maintained. 
Correct  information  on  the  location  of  the  data  and  who  should  be  on 
the  audit  roster  needs  to  be  communicated  to  the  auditor. 

•  Develop  a  plan  for  effort  data  organization  such  that  short  term  (ease 
of  entry)  and  long  term  (combining  data  for  overall  comparison 
purposes)  goals  are  considered. 

•  Develop  questions  and  metrics  to  measure  accomplishments  in 
response  to  the  business  goal  to  “improve  the  project  planning 
process.” 

•  Archive  all  project  planning  work  products  using  the  organization’s 
designated  CM  tool. 
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•  Conduct  a  formal  audit  on  the  project  planning  process  at 
during  project  execution. 

•  Update  policy,  process,  and  procedure  documentation 
additional  activities. 


least  once 


to  reflect 
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VI.  PROJECT  MONITORING  AND  CONTROL  STUDY 


The  results  and  analysis  of  the  project  monitoring  and  control  process  for 
the  two  case  studied  are  shown  in  the  following  subsections.  An  outline  of  the 
goals  and  associated  practices  defined  for  the  CMMI  Project  Monitoring  and 
Control  Process  Area,  a  score  for  each  project  studied  (Case  1  and  Case  2),  and 
justification  for  each  rating  is  first  provided.  A  general  analysis  of  the  evaluation 
results  for  each  case  follows  the  data  presentation.  The  analysis  includes 
presentation  and  comparison  to  the  post-mortem  lessons  learned  to  determine  if 
there  are  correlations  with  the  evaluation  results.  Note  that  the  lessons  are 
unique  to  the  individual  projects,  and  that  only  the  lessons  that  apply  to  the  area 
of  study  are  included.  Finally,  a  comparison  of  the  two  cases  is  made  and  a  list 
of  suggested  improvements  is  provided. 

A.  CASE  1  AND  CASE  2  EVALUATION  RESULTS 

The  goals  and  associated  practices  defined  for  the  CMMI  Project 
Monitoring  and  Control  Process  Area,  and  individual  practice  ratings  assigned  by 
the  project  team  are  as  follows: 

•  Specific  Goal  (SG)  1  -  “Actual  performance  and  progress  of  the  project 
are  monitored  against  the  project  plan.” 

o  Specific  Practice  (SP)  1.1  -  “Monitor  (after  documenting  initial 
estimates,  changed  estimates  if  project  parameters  changed) 
actual  values  of  the  project  planning  parameters  against  the  project 
plan”  was  “Performed  Adequately  for  both  Case  1  and  Case  2. 
Note:  “Attributes”  are  “inputs  to  estimating  models,  such  as 
complexity,  or  skills/effort,  or  previous  estimates  of  the  software 
changes”  (i.e.,  values  that  affect  calculation  of  the  size  of  a  project). 

1.  The  Case  1  project  schedule,  effort,  and  scope  were 
identified,  reviewed,  and  baselined  (approved  by  senior 
management)  during  planning  stages  of  the  project. 
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Subsequent  review  and  revision  of  this  data  took  place  at 
regular  project  and  senior  management  review  meetings. 

2.  The  Case  2  project  schedule,  effort,  and  scope  were 
identified  and  reviewed  during  planning  stages  of  the  project. 
However,  they  were  never  baselined  (approved  by  senior 
management).  Subsequent  review  and  revision  of  this  data 
took  place  at  regular  project  and  senior  management  review 
meetings.  However,  this  activity  dropped  off  early  in  the 
project  as  the  leadership  in  the  project  engineer  (manager) 
role  also  dropped  off.  The  project  team  was  able  to  make 
planned  scope  deliveries  within  schedule,  cost  anyway. 

3.  For  both  projects,  project  plans  were  based  on  parameters. 
Resources  and  delivery  contents  are  allocated,  and  affected 
by,  actual  attributes  (i.e.,  problem  analysis  may  reveal  a 
lesser  or  greater  actual  complexity  factor). 

o  SP1.2  -  “Monitor  commitments  against  those  identified  in  the 
project  plan”  was  “Performed  Adequately”  for  both  Case  1  and 
Case  2. 

1.  Case  1  project  commitments  (internal  and  external)  to 
scope,  schedule,  and  cost  were  reviewed  regularly  at 
internal  project  and  senior  review  management  review 
meetings,  and  external  overarching  integrated  product  team 
meetings.  This  demonstration  is  evident  from  copies  of 
associated  minutes  and  presentation  materials. 

2.  Case  2  external  commitments  were  reviewed  with  industry 
partners  on  a  regular  basis.  Internal  commitments  were 
reviewed  initially  at  project  and  senior  management  review 
meetings.  However,  this  activity  dropped  off  early  in  the 
project  as  the  leadership  in  the  project  engineer  (manager) 
role  also  dropped  off.  The  project  team  was  able  to  make 
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planned  scope  deliveries  within  schedule  and  cost  anyway 
since  commitments  were  reviewed  with  the  software  and  test 
teams,  even  though  it  was  a  less  formal  process. 

3.  Also,  planned  SQA  audits  were  conducted  for  both  projects, 
in  addition  to  monitoring  peer  review  activity, 
o  SP1.3  -  “Monitor  risks  against  those  identified  in  the  project  plan” 
was  “Performed  Adequately  for  Case  1  and  “Not  Performed 
Adequately”  for  Case  2. 

1.  Case  1  risks  were  identified,  reviewed,  and  baselined 
(approved  by  senior  management)  during  planning  stages  of 
the  project.  A  separate  Risk  Management  Plan  was 
prepared.  Subsequent  review  and  revision  of  this  data  took 
place  at  regular  project  and  senior  management  review 
meetings. 

2.  Case  2  risks  were  identified  and  reviewed  during  planning 
stages  of  the  project,  as  a  Risk  Management  section  was 
included  in  the  project  plan.  Subsequent  review  and  revision 
of  this  data  took  place  at  regular  project  and  senior 
management  review  meetings.  However,  this  activity 
dropped  off  early  in  the  project  as  the  leadership  in  the 
project  engineer  (manager)  role  also  dropped  off.  The 
project  team  was  able  to  make  planned  scope  deliveries 
within  schedule  and  cost  anyway.  However,  the  project 
team  indicated  that  monitoring  risks  informally  was  not 
sufficient. 

o  SP1.4  -  “Monitor  the  management  of  (internal)  project  data 
(schedule,  cost  in  terms  of  effort,  and  scope)  against  the  project 
plan”  was  “Performed  Adequately  for  both  Case  1  and  Case  2. 
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1.  Project  data  was  interpreted  as  “documents  required  to 
support  the  project  such  as  work  products,  action  item  lists, 
meeting  minutes,  or  lessons  learned  reports.” 

2.  For  both  projects,  the  plan  to  manage  project  data  was 
provided  in  the  project  plan.  It  included  directions  for 
generation  and  distribution  of  both  deliverable  and  non¬ 
deliverable  work  products,  CM  (SCR  database),  and  QA 
artifacts.  Meeting  minutes,  QA  audit  results,  and  the  project 
action  item  list  served  as  records  used  to  monitor  the 
management  of  project  data. 

3.  The  Case  1  project  schedule,  effort,  and  scope  were 

identified,  reviewed,  and  baselined  (approved  by  senior 
management)  during  planning  stages  of  the  project. 

Subsequent  review  and  revision  of  this  data  took  place  at 

regular  project  and  senior  management  review  meetings. 

4.  The  Case  2  project  schedule,  effort,  and  scope  were 

identified  and  reviewed  during  planning  stages  of  the  project. 
However,  they  were  never  baselined  (approved  by  senior 
management).  Subsequent  review  and  revision  of  this  data 
took  place  at  regular  project  and  senior  management  review 
meetings.  However,  this  activity  dropped  off  early  in  the 
project  as  the  leadership  in  the  project  engineer  (manager) 
role  also  dropped  off.  The  project  team  was  able  to  make 
planned  scope  deliveries  within  schedule  and  cost  anyway. 

o  SP1 .5  -  “Monitor  stakeholder  involvement  against  the  project  plan” 
was  “Performed  Adequately  for  both  Case  1  and  Case  2. 

1.  Case  1  stakeholder  involvement  was  identified,  reviewed, 
and  baselined  (approved  by  senior  management)  during 
planning  stages  of  the  project.  Subsequent  review  and 
revision  of  this  data  took  place  at  regular  project  and  senior 
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management  review  meetings,  overarching  integrated 
product  team  or  software  meetings  with  the  industry  partner. 

2.  Case  2  stakeholder  involvement  was  identified  and  reviewed 
during  planning  stages  of  the  project.  However,  they  were 
never  baselined  (approved  by  senior  management). 
Subsequent  review  and  revision  of  this  data  took  place  at 
regular  project  and  senior  management  review  meetings, 
overarching  integrated  product  team  or  software  meetings 
with  the  industry  partner.  However,  internal  senior 
management  review  and  project  meeting  activities  dropped 
off  early  in  the  project  as  the  leadership  in  the  project 
engineer  (manager)  role  also  dropped  off.  The  project  team 
was  able  to  make  planned  scope  deliveries  within  schedule 
and  cost  anyway.  External  stakeholders  were  involved,  and 
when  there  was  a  schedule  misunderstanding,  it  was  an 
exception  (it  was  not  typical  for  the  industry  partner  not  to 
have  informed  the  organization  of  a  critical  change), 
o  SP1.6  -  “Periodically  review  the  project’s  progress,  performance, 
and  issues”  was  “Performed  Adequately”  for  both  Case  1  and 
Case  2. 

1.  Case  1  project  progress,  performance,  and  issues  were 
identified  (in  terms  of  software  drops),  reviewed,  and 
baselined  (approved  by  senior  management)  during  planning 
stages  of  the  project.  Subsequent  review  and  revision  of  this 
data  took  place  at  regular  project  and  senior  management 
review  meetings. 

2.  Case  2  project  progress,  performance,  and  issues  were 
identified  (in  terms  of  software  drops)  and  reviewed  during 
planning  stages  of  the  project  but  never  baselined  (approved 
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by  senior  management).  Subsequent  review  and  revision  of 
this  data  took  place  at  regular  project  and  senior 
management  review  meetings  but  this  activity  dropped  off 
early  in  the  project  as  the  leadership  in  the  project  engineer 
(manager)  role  also  dropped  off.  The  project  team  was  able 
to  make  planned  scope  deliveries  within  schedule,  cost 
anyway.  The  project  team  began  to  use  meetings  with 
industry  partner  and  internal  “change  control  board” 
meetings  as  a  forum  to  exchange  project  information.  While 
the  process  was  less  formal,  it  was  enough, 
o  SP1.7  -  “Review  the  accomplishments  and  results  of  the  project  at 
selected  project  milestones”  was  “Performed  Adequately  for  both 
Case  1  and  Case  2. 

1.  “Milestones”  was  interpreted  as  “software  releases”  for  the 
organization. 

2.  Case  1  project  progress,  performance,  and  issues  were 
identified  (in  terms  of  software  drops),  reviewed,  and 
baselined  (approved  by  senior  management)  during  planning 
stages  of  the  project.  Subsequent  review  and  revision  of  this 
data  took  place  at  regular  project  and  senior  management 
review  meetings.  In  addition,  planned  product  audits  were 
conducted  following  the  delivery  of  each  software  drop 
(including  QA  build/functional/configuration/test  tool,  and 
change  control  board  audits).  In  addition,  the  software  and 
test  leads  review  product  release  notes  to  ensure  that  the 
results  of  change  control  board  activities  were  reflected. 

3.  Case  2  project  progress,  performance,  and  issues  were 
identified  (in  terms  of  software  drops)  and  reviewed  during 
planning  stages  of  the  project  but  never  baselined  (approved 
by  senior  management).  Subsequent  review  and  revision  of 
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this  data  took  place  at  regular  project  and  senior 
management  review  meetings.  However,  this  activity 
dropped  off  early  in  the  project  as  the  leadership  in  the 
project  engineer  (manager)  role  also  dropped  off.  The 
project  team  was  able  to  make  planned  scope  deliveries 
within  schedule,  cost  anyway.  The  project  team  began  to 
use  meetings  with  the  industry  partner  and  internal  “change 
control  board”  meetings  as  a  forum  to  exchange  project 
information.  While  the  process  was  less  formal,  it  was 
enough. 

•  Specific  Goal  (SG)  2  -  “Corrective  actions  are  managed  to  closure 
when  the  project’s  performance  or  results  deviate  significantly  from  the 
plan.” 

o  SP2.1  -  “Correct  and  analyze  the  issues  and  determine  the 
corrective  actions  necessary  to  address  the  issues”  was 
“Performed  Adequately”  for  both  Case  1  and  Case  2. 

1.  Note  that  the  goal  (SG  2)  refers  to  the  project  plan  versus 
actual  performance. 

2.  Issues  were  identified,  assigned  (resource  and  expected 

closure  date),  and  logged  on  the  Case  1  project  action  item 
list.  Team  members  would  use  the  project  or  senior 

management  review  meetings  to  make  collective  decisions 
on  how  to  resolve  issues.  Subsequent  review  and  revision 
of  this  data  took  place  at  regular  project  and  senior 
management  review  meetings 

3.  Issues  were  identified,  assigned  (resource  and  expected 

closure  date),  and  logged  on  the  Case  2  project  action  item 
list.  Team  members  would  use  the  project  or  senior 

management  review  meetings  to  make  collective  decisions 
on  how  to  resolve  issues. 
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4.  Subsequent  review  and  revision  of  this  data  took  place  at 
regular  project  and  senior  management  review  meetings. 
However,  this  activity  dropped  off  early  in  the  project  as  the 
leadership  in  the  project  engineer  (manager)  role  also 
dropped  off.  The  process  became  less  formal,  and  the 
project  team  was  still  able  to  make  planned  scope  deliveries 
within  schedule,  cost  anyway.  Meetings  with  the  industry 
partner  and  internal  change  control  board  meetings  were 
used  as  forums  to  exchange  project  information, 
o  SP2.2  -  “Take  corrective  action  on  identified  issues”  was 
“Performed  Adequately”  for  both  Case  1  and  Case  2. 

1.  Because  of  the  activities  conducted  (identified  above), 
issues  got  resolved.  Both  projects  had  action  item  lists  to 
demonstrate  that  corrective  action  was  taken, 
o  SP2.3  -  “Manage  corrective  actions  to  closure”  was  “Performed 
Adequately”  for  both  Case  1  and  Case  2. 

1 .  Because  of  the  activities  conducted  (identified  above), 
issues  got  resolved.  Both  projects  had  action  item  list’s  to 
demonstrate  that  corrective  action  was  taken,  and  closure 
reached. 

•  Generic  Goal  (GG)  2  -  “The  process  is  institutionalized  as  a  managed 
process.” 

o  Generic  Practice  (GP)  2.1  -  “Establish  and  maintain  an 

organizational  policy  for  planning  and  performing  the  Project 
Monitoring  and  Control”  process  was  “Performed  Adequately”  for 
both  Case  1  and  Case  2. 

1.  The  organization  has  an  established  “Software  Project 
Tracking  and  Oversight  Policy.” 
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o  GP2.2  -  “Establish  and  maintain  the  plan  for  performing  the  Project 
Monitoring  and  Control  process”  was  “Performed  Adequately”  for 
Case  1  and  “Not  Performed  Adequately^’  for  Case  2. 

1.  “Establish  and  maintain”  implies  “baselining”,  which  for  the 
organization  translates  into  senior  management  approval 
(sign-off). 

2.  A  plan  for  monitoring  and  controlling  the  Case  1  project 
schedule,  effort,  and  scope  were  established  in  the  project 
plan,  which  was  reviewed  and  baselined  (approved  by  senior 
management)  early  in  the  project.  The  plan  did  not  change. 

3.  A  plan  for  monitoring  and  controlling  the  Case  2  project 
schedule,  effort,  and  scope  were  established  in  the  project 
plan,  which  was  reviewed  early  in  the  project  but  never 
baselined  (approved  by  senior  management).  The  plan  did 
not  change. 

o  GP2.3  -  “Provide  adequate  resources  for  performing  the  Project 
Monitoring  and  Control  process,  developing  the  work  products,  and 
providing  the  services  of  the  process”  was  “Performed 
Adequately”  for  Case  1  and  “Not  Performed  Adequately ”  for 
Case  2. 

1 .  “Resources”  could  be  persons  (such  as  the  project  manager) 
or  tools  (such  as  the  effort  worksheets,  action  item  lists,  or 
MS  project) 

2.  Senior  management  provided  the  Case  1  project 
management  resource  to  monitor  and  report  on  the  project’s 
progress,  performance,  and  issues  related  to  the  schedule, 
effort,  and  scope. 

3.  Senior  management  provided  the  Case  2  project 
management  resource  to  monitor  and  report  on  the  project’s 
progress,  performance,  and  issues  related  to  the  schedule, 
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effort,  and  scope.  Other  priorities  took  precedence  and  it 
became  necessary  to  identify  a  new  person  to  fulfill  this  role. 
Delays  in  this  decision,  and  other  priorities,  caused 
significant  lack  in  these  activities. 

o  GP2.4  -  “Assign  responsibility  and  authority  for  performing  the 
process,  developing  the  work  products,  and  providing  the  services 
of  the  Project  Monitoring  and  Control  process”  was  “Performed 
Adequately”  for  Case  1  and  “Not  Performed  Adequately  for 
Case  2. 

1.  Senior  management  provided  the  Case  1  project 

management  resource  to  monitor  and  report  on  the  project’s 
progress,  performance,  and  issues  related  to  the  schedule, 
effort,  and  scope. 

2.  Senior  management  provided  the  Case  2  project 

management  resource  to  monitor  and  report  on  the  project’s 
progress,  performance,  and  issues  related  to  the  schedule, 
effort,  and  scope.  Other  priorities  took  precedence  and  it 
became  necessary  to  identify  a  new  person  to  fulfill  this  role. 
Delays  in  this  decision,  and  other  priorities,  caused 

significant  lack  in  these  activities. 

o  GP2.5  -  “Train  the  people  performing  or  supporting  the  Project 
Monitoring  and  Control  process”  as  needed  was  “Performed 
Adequately”  for  both  Case  1  and  Case  2. 

1.  Training  consisted  of  previous  experience,  individual 
education,  and  on  the  job  training  for  both  the  Case  1  and 
Case  2  project  teams.  The  project  team  indicated  that  the 
training  was  adequate  to  accomplish  Project  Monitoring  and 
Control  goals. 
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o  GP2.6  -  “Place  designated  work  products  of  the  Project  Monitoring 
and  Control  process  under  appropriate  levels  of  configuration”  was 
“Not  Performed  Adequately”  for  Case  1  or  Case  2. 

1.  The  project  monitoring  and  control  work  products  were 
identified  as  configuration  items  in  both  Case  1  and  Case  2 
project  plans,  and  were  placed  under  CM  in  accordance  with 
the  draft.  While  the  SPMP  was  not  approved,  the  work  was 
done.  The  NPA  rating  was  given  instead  of  the  PA  rating 
since  the  products  are  stored  in  the  NextGen  shared  folders 
(not  in  ClearCase  -  our  designated  CM  tool), 
o  GP  2.7  -  “Identify  and  involve  the  relevant  stakeholders  of  the 
Project  Monitoring  and  Control  process”  as  planned  was 
“Performed  Adequately  for  Case  1  and  “Not  Performed 
Adequately”  for  Case  2. 

1.  Case  1  project  progress,  performance,  and  issues  were 
identified,  reviewed,  and  baselined  (approved  by  senior 
management)  during  planning  stages  of  the  project. 
Subsequent  review  and  revision  of  this  data  took  place  at 
regular  project  and  senior  management  review  meetings. 
External  stakeholders  were  identified  and  regularly  involved 
as  planned. 

2.  Case  2  project  progress,  performance,  and  issues  were 
identified  and  reviewed  during  planning  stages  of  the  project 
but  never  baselined  (approved  by  senior  management). 
Subsequent  review  and  revision  of  this  data  took  place  at 
regular  project  and  senior  management  review  meetings. 
However,  this  activity  dropped  off  early  in  the  project  as  the 
leadership  in  the  project  engineer  (manager)  role  also 
dropped  off.  The  project  team  was  able  to  make  planned 
scope  deliveries  within  schedule,  cost  anyway.  External 
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stakeholders  were  identified  and  regularly  involved  as 
planned. 

o  GP2.8  -  “Monitor  and  control  the  Project  Monitoring  and  Control 
process  against  the  plan  for  performing  the  process  and  take 
appropriate  corrective  action”  was  “Not  Performed  Adequately1’ 
for  both  Case  1  and  Case  2. 

1.  The  process  was  defined  in  the  project  plans  for  both 
projects,  but  reports  of  the  data  did  not  materialize  as  often 
planned,  nor  were  they  as  complete  as  planned.  No 
changes  in  the  process  or  corrective  action  resulted  from  this 
omission. 

o  GP2.9  -  “Objectively  evaluate  adherence  of  the  Project  Monitoring 
and  Control  process  against  its  process  description,  standards,  and 
procedures,  and  address  noncompliance”  was  “Not  Performed 
Adequately”  for  both  Case  1  and  Case  2. 

1.  The  organization’s  overarching  process  for  SQA  discusses 
independently  evaluating  the  project  monitoring  and  control 
process.  This  was  done  by  SQA  at  senior  management 
review  meetings  (more  often  for  Case  1  than  for  Case  2). 
However,  since  an  independent  formal  audit  was  not 
performed  for  either  project,  the  lesser  rating  was  assigned. 

o  GP2.10  -  “Review  the  activities,  status,  and  results  of  the  Project 
Monitoring  and  Control  process  with  higher-level  management  and 
resolve  issues”  was  “Performed  Adequately  for  Case  1  and  “Not 
Performed  Adequately”  for  Case  2. 

1.  Case  1  project  progress,  performance,  and  issues  were 
identified,  reviewed,  and  baselined  (approved  by  senior 
management)  during  planning  stages  of  the  project. 
Subsequent  review  and  revision  of  this  data  took  place  at 
regular  project  and  senior  management  review  meetings. 
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External  stakeholders  were  identified  and  regularly  involved 
as  planned. 

2.  Case  2  project  progress,  performance,  and  issues  were 
identified  and  reviewed  during  planning  stages  of  the  project 
but  never  baselined  (approved  by  senior  management). 
Subsequent  review  and  revision  of  this  data  took  place  at 
regular  project  and  senior  management  review  meetings. 
However,  this  activity  dropped  off  early  in  the  project  as  the 
leadership  in  the  project  engineer  (manager)  role  also 
dropped  off.  The  project  team  was  able  to  make  planned 
scope  deliveries  within  schedule,  cost  anyway.  External 
stakeholders  were  identified  and  regularly  involved  as 
planned. 

•  Generic  Goal  (GG)  3  -  “The  process  is  institutionalized  a  defined 
process.” 

o  GP  3.1  -  “Establish  and  maintain  the  description  of  a  defined  Project 
Monitoring  and  Control  process”  was  “Performed  Adequately  for 
both  Case  1  and  Case  2. 

1.  The  following  organization  procedures  have  been  established, 
and  have  not  changed  during  the  time  that  both  Case  1  and  Case 
2  projects  were  executed:  “Effort  Tracking  Procedure”,  “Formal 
Reviews  at  Milestones  Procedure”,  “Development  Hours  Tracking 
Form”,  “Verification  Hours  Tracking  Form”,  and  “Weekly  One- 
Liners  Report  Form” 

o  GP  3.2  -  “Collect  work  products,  measures,  measurement  results, 
and  improvement  information  derived  from  planning  and  performing 
the  Project  Monitoring  and  Control  process  to  support  the  future  use 
and  improvement  of  the  organization’s  processes  and  process  assets” 
was  “Not  Performed  Adequately  for  both  Case  1  and  Case  2. 

o 
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B.  CASE  1  ANALYSIS 


1.  Analysis  of  Evaluation  Results 

The  following  observations  were  made  based  on  process  evaluation 
ratings  given: 

•  100%  of  the  practices  were  performed  to  some  degree  (all  were  rated 
either  “Not  Performed  Adequately”,  or  “Performed  Adequately”  -  none 
were  rated  “Not  Performed”).  All  practices  are  being  performed  to 
some  extent. 

•  Overall,  18/22  or  82%  of  the  practices  were  rated  “Performed 
Adequately”.  This  describes  the  success  of  most  practices  associated 
with  each  of  the  three  goals  in  this  Process  Area  indicating  that  the 
improvements  will  not  need  to  be  concentrated  on  one  particular  goal, 
rather,  a  few  practices  in  each  area  will  need  to  be  improved. 

2.  Lessons  Learned  Reported 

a.  The  Initial  Set  of  Identified  Risks  Was  Not  Effectively 
Tracked 

New  risks  were  not  effectively  elicited  from  the  full  project  staff. 
Some  staff  members  were  not  aware  of  the  risks  being  tracked.  Training  in  the 
risk  management  process  should  be  provided  to  ensure  that  true  risks  to  project 
completion  are  identified,  assessed,  prioritized,  mitigated,  and  that  contingency 
plans  are  in  place  should  a  risk  be  realized.  The  current  set  of  risks  should  be 
available  to  all  project  staff  in  a  shared  repository.  For  the  Case  1  project,  not  all 
of  the  initial  risks  were  true  risks  that  could  affect  project  completion.  The  initial 
set  of  identified  risks  was  not  effectively  tracked.  New  risks  were  not  effectively 
elicited  from  the  full  project  staff.  Some  staff  members  were  not  aware  of  the 
risks  being  tracked. 
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b.  More  Complicated  Problems  ere  Being  Resolved,  But  the 
Effort  Audits  Didn’t  Show  How  Much  Effort  Was  Expended 
On  Each  Problem 

There  should  be  more  detail  recorded  in  planning  and  estimating 
resource  requirements.  For  Case  1,  hours  estimates  were  not  done  over  time 
intervals.  Instead,  there  was  just  one  big  lump  sum.  More  complicated  problems 
were  being  resolved,  but  the  effort  audits  didn’t  show  how  much  effort  was 
expended  on  each  problem.  With  other  projects  competing  for  human  resources, 
we  need  to  keep  in  mind  that  the  project  can  lose  people. 

c.  One  Large  All-Encompassing  Schedule  Became  Too 
Complex  and  Unwieldy 

The  project  should  be  tracked  at  two  levels  -  project  and  functional. 
All  of  the  schedules  should  be  posted  on  a  shared  repository  so  that  all  project 
staff  are  aware  of  key  dates.  For  the  Case  1  project,  an  initial  attempt  to  track 
progress  on  one  large  all-encompassing  schedule  became  too  complex  and 
unwieldy.  Instead,  the  project  was  tracked  by  the  Project  Lead  to  a  joint  industry 
partner  and  organization  schedule.  The  functional  leads  (software,  test,  SCM, 
and  SQA)  tracked  the  project  at  the  software  requirement  level. 

d.  There  Was  a  Significant  Lack  of  Compliance  in  Recording 
Effort  Data 

Provide  feedback  to  the  project  team  on  the  definition,  collection, 
and  use  of  metrics.  For  example,  the  project  collected  the  hours  expended  by 
each  member  of  the  project  team.  In  order  to  do  this  effectively,  each  functional 
area  needs  well-defined  categories  in  which  to  record  their  hours.  The  method 
and  frequency  of  collecting  these  records  needs  to  be  fully  explained  to  ensure 
they  are  up-to-date.  Finally,  the  reason  for  collecting  hours  and  how  this 
information  will  be  used  also  needs  to  be  fully  explained.  Similar  efforts  are 
needed  for  the  remaining  project  metrics  in  order  to  improve  compliance  by  the 
project  staff. 
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e.  The  Project  Team  Was  Not  Well  Informed  With  a  Summary  of 
Senior  Management’s  Comments  and  Directives 

Provide  feedback  to  the  project  staff  on  the  results  of  the  Senior 
Management  Reviews  (SMRs).  The  SMRs  were  effective  in  presenting  the  status 
of  the  project  to  the  Associate  Director  (AD)  and  getting  his  directives  for 
resolving  management  issues.  However,  only  functional  area  leads  attended  the 
SMRs.  The  remaining  project  staff  should  have  the  option  to  attend.  In  addition, 
while  the  slides  presented  at  the  SMR  were  made  available  to  all,  SMR  minutes 
were  not  published.  This  missed  the  opportunity  to  keep  the  project  staff 
informed  with  a  summary  of  senior  management  comments  and  directives. 

f.  Individual  Project  Team  Members  Did  Not  Apply  Their  Own 

Experience  to  Help  in  Identifying,  Assessing,  Prioritizing, 
Mitigating,  and  Tracking  Risks 

All  members  of  the  project  team  need  training  in  risk  management. 
Everyone  needs  to  be  involved  in  identifying,  assessing,  prioritizing,  mitigating, 
and  tracking  risks  and  in  preparing  contingency  plans  in  case  a  particular  risk  is 
realized.  Each  member  should  be  capable  of  applying  their  own  experience  to 
assist  the  Project  Lead  in  identifying  and  managing  risks.  This  will  help  ensure 
that  all  risks  that  could  significantly  impact  a  project  are  identified. 

g.  Project  Risks  Were  Not  Presented  as  Effectively  as  They 
Could  Have  Been 

The  format  for  presenting  the  highest  priority  risks  at  Senior 
Management  Reviews  should  effectively  identify  issues  for  senior  management 
consideration.  Various  formats  were  tried. 

h.  SCM  Was  Not  Viewed  as  a  Continuous  Process  Involving 
Participation  by  the  Entire  Project  Staff 

SCM  is  not  just  a  repository  for  project  work  products  but  a 
continuous  process  involving  participation  by  the  entire  project  staff.  Such 
participation  should  lead  to  improvements  in  the  process  as  staff  members 
generate  Process  Change  Requests  (PCRs)  to  resolve  issues. 

/.  Some  Metrics,  Such  As  Effort  Data,  Were  Difficult  To  Collect 
Manually  and  Presented  Problems  in  the  Staff  Not 
Understanding  the  Benefit  of  Their  Use 
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Select  project  metrics  for  the  information  they  provide  and  ensure 
their  ease  of  collection.  The  project  staff  was  not  always  aware  of  what  metrics 
were  collected  and  what  they  would  be  used  for.  Some  metrics,  such  as  effort 
data,  was  difficult  to  collect  manually.  In  addition,  it  was  difficult  to  get  all  project 
team  members  to  record  data.  Perhaps  this  was  because  they  did  not 
understand  the  benefits  of  collecting  this  data. 

C.  CASE  2  ANALYSIS 

1.  Analysis  of  Evaluation  Results 

The  following  observations  were  made  based  on  process 

evaluation  ratings  given: 

•  100%  of  the  practices  were  performed  to  some  degree  (all  were  rated 
either  “Not  Performed  Adequately”,  or  “Performed  Adequately”  -  none 
were  rated  “Not  Performed”).  All  practices  are  being  performed  to 
some  extent. 

•  Overall,  12/22  or  55%  of  the  practices  were  rated  “Performed 
Adequately”.  This  describes  the  success  of  most  practices  associated 
with  each  of  the  three  goals  in  this  Process  Area  indicating  that  the 
improvements  will  not  need  to  be  concentrated  on  one  particular  goal, 
rather,  a  few  practices  in  each  area  will  need  to  be  improved. 

2.  Lessons  Learned  Reported 

a.  Schedules  and  Key  Project  Artifacts  Were  Not  Kept  Up  to 
Date  and  Visible 

Schedules  and  key  project  artifacts  were  not  presented  at  every 
meeting,  nor  were  responsibilities  for  maintenance  assigned.  In  addition,  there 
were  no  defined  standard  access  rules  for  these  artifacts. 

b.  Elements  of  the  Plan  Were  Not  Performed 

Project  meetings,  senior  management  review  meetings,  and  risk 
management  activities  were  not  executed.  Plans  were  not  coordinated  and 
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action  items  were  not  recorded  or  tracked  to  closure.  The  project  lacked  a  full 
time  project  engineer  (manager)  to  lead  these  efforts. 

c.  Project  Metrics  Not  Accurate  and  are  Not  Being  Used 

Essential  project  data  was  not  defined  and  did  not  have 

spreadsheets  to  manipulate  and  store  the  data,  did  not  have  built-in  reporting 
requirements  for  project  meetings  and  senior  management  reviews.  The  data 
collection  process  needs  to  be  reviewed  and  refined  for  ease  and  efficiency. 

d.  An  Overall  Project  Schedule  Was  Non-Existent 

The  lack  of  a  project  schedule  resulted  in  differing  milestones 
between  the  developers,  testers,  CM,  SQA,  and  the  industry  partner. 

e.  Lack  of  Project  Manager 

The  lack  of  a  project  lead  caused  much  frustration  due  to  the  lack 
of  coordination,  both  internal  and  external. 

f.  Lack  of  Commitment  to  Effort  Recording/Reporting 

There  were  a  variety  of  problems  with  actual  effort  data  recorded. 

There  was  much  missing  data,  and  estimates  made  well  past  actual  work  done 
were  likely  less  accurate.  In  addition,  QA  audits  sometimes  (mistakenly) 
included  team  members  who  were  no  longer  working  on  the  project,  and  team 
member  response  did  not  seem  to  greatly  improve  over  time.  It  is  now 
impossible  to  compare  final  estimates  to  actual  time  spent. 

g.  There  Was  a  Lot  of  Missing  Data  and  Much  Data  is 
Recorded  a  Considerable  Amount  of  Time  After  Actual  Time 
is  Spent 

There  appeared  to  be  a  lot  of  missing  data,  and  much  data  is 
recorded  a  considerable  amount  of  time  after  actual  time  is  spent.  Either  data  is 
lost  or,  when  finally  recorded,  may  be  less  accurate  than  it  could  have  been  if 
recorded  on  a  more  regular  basis.  Only  33%  of  the  Case  2  project  team  has 
updated  their  worksheets  within  the  last  month. 
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h.  It  Has  Been  Difficult  to  Determine  if  Worksheets  are  Current 
(and  “Zero”  Time  Has  Been  Put  Towards  the  Project)  or  if 
Data  Entry  is  Lagging 

For  audit  purposes,  the  latter  was  assumed.  If  correct  information 

is  not  communicated  to  the  auditor,  audit  results  may  be  skewed. 

/.  There  is  Not  a  Consistent  Naming  Convention  Used  to 
Identify  the  Effort  Record  Worksheets 

QA  worksheets  use  fiscal  year  and  team  member  identification  in 

the  file  names.  Test  group  worksheets  use  only  name  for  Case  2  worksheets 

and  name  and  project  identification  for  Case  2  worksheets  in  the  file  names. 

Development  and  lab  worksheets  use  only  name  identification  in  the  file  names. 

Once  archived,  it  would  be  difficult  to  determine  what  project  the  data  was 

recorded  for  with  out  opening  each  file. 

D.  COMPARISON  OF  CASE  1  TO  CASE  2 

1.  Goals 

The  goals  associated  with  the  PMC  PA  are  as  follows: 

•  Specific  Goal  (SG)  1  -  Specific  Goal  (SG)  1  -  “Actual  performance  and 
progress  of  the  project  are  monitored  against  the  project  plan.” 

•  Specific  Goal  (SG)  2  -  “Corrective  actions  are  managed  to  closure 
when  the  project’s  performance  or  results  deviate  significantly  from  the 
plan.” 

•  Generic  Goal  (GG)  2  -  “The  process  is  institutionalized  as  a  managed 
process.” 

•  Generic  Goal  (GG)  3  -  “The  process  is  institutionalized  a  defined 
process.” 

The  achievement  by  Case  1  and  Case  2  in  terms  of  goal  success  can  be 
described  by  a  comparison  of  the  number  of  practices  given  the  “Performed 
Adequately”  rating  to  the  total  number  of  practices  for  that  goal.  The  results  of 
this  analysis  follow: 
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Case  1  -  SG  1  -  7/7  or  100% 

SG  2  -  3/3  or  100% 

GG  2 -7/10  or  70% 

GG  3-  1/2  or  50% 

Case  2  -  SG  1  -  6/7  or  86% 

SG  2-3/3  or  100% 

GG  2-2/10  or  20% 

GG  3-  1/2  or  50% 

No  overall  goal  improvements  can  be  demonstrated  by  this  data,  only 
digression  for  SG  1 ,  and  GG  2. 


2.  Practices 


Practice 

Case  1 

Case  2 

Not 

Performed 

Adequately 

Performed 

Adequatel 

y 

Not 

Performed 

Adequately 

Performed 

Adequately 

SP  1.1 

X 

X 

SP  1.2 

X 

X 

SP  1.3 

X 

X 

SP  1.4 

X 

X 

SP  1.5 

X 

X 

SP  1.6 

X 

X 

SP  1.7 

X 

X 

SP  2.1 

X 

X 

SP  2.2 

X 

X 

SP  2.3 

X 

X 

GP  2.1 

X 

X 

GP  2.2 

X 

X 

GP  2.3 

X 

X 

GP  2.4 

X 

X 

GP  2.5 

X 

X 

GP  2.6 

X 

X 

GP  2.7 

X 

X 

GP  2.8 

X 

X 

GP  2.9 

X 

X 

GP  2.10 

X 

X 

GP  3.1 

X 

X 

GP  3.2 

X 

X 

Table  3  Summary  of  Project  Monitoring  and  Control  Evaluation  Results 

76 


The  results  do  not  demonstrate  improvement  from  Case  1  to  Case  2  in 
any  project  monitoring  and  control  practice  evaluated.  They  do  show  digression 
in  the  following  practices  (highlighted  rows): 

•  SP1 .3  -  Monitor  risks  against  those  identified  in  the  project  plan.” 

•  GP2.2  -  Establish  and  maintain  the  plan  for  performing  the  Project 

Monitoring  and  Control  process. 

•  GP2.3  -  Provide  adequate  resources  for  performing  the  Project 

Monitoring  and  Control  process,  developing  the  work  products,  and 
providing  the  services  of  the  process. 

•  GP2.4  -  Assign  responsibility  and  authority  for  performing  the  process, 

developing  the  work  products,  and  providing  the  services  of  the  Project 
Monitoring  and  Control  process. 

•  GP  2.7  -  Identify  and  involve  the  relevant  stakeholders  of  the  Project 

Monitoring  and  Control  process. 

•  GP  2.10  -  Review  the  activities,  status,  and  results  of  the  Project 

Monitoring  and  Control  process  with  higher-level  management  and 
resolve  issues. 

In  addition,  results  indicate  that  for  both  projects,  the  following  practices 
were  “Not  Performed  Adequately:’’ 

•  GP2.6  -  Place  designated  work  products  of  the  Project  Monitoring  and 
Control  process  under  appropriate  levels  of  configuration 
management. 

•  GP2.8  -  Monitor  and  control  the  Project  Monitoring  and  Control 
process  against  the  plan  for  performing  the  process  and  take 
appropriate  corrective  action. 

•  GP2.9  -  Objectively  evaluate  adherence  of  the  Project  Monitoring  and 
Control  process  against  its  process  description,  standards,  and 
procedures,  and  address  noncompliance. 
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•  GP  3.2  -  Collect  work  products,  measures,  measurement  results,  and 
improvement  information  derived  from  planning  and  performing  the 
Requirements  Management  process  to  support  the  future  use  and 
improvement  of  the  organization’s  processes  and  process  assets. 

3.  Comparison  of  Evaluation  Results  to  Lessons  Learned 

Examination  of  the  comments  from  the  evaluation  results  for  which  both 
projects  had  “Not  Performed  Adequately”  rating  show  that  designated  work 
products  were  not  put  under  appropriate  levels  of  configuration  management,  the 
PMC  process  was  not  monitored,  and  corrective  action  was  not  taken  when  it 
was  needed.  Similar  to  other  process  areas  evaluated,  some  metrics  were 
collected,  some  artifacts  were  stored,  and  some  lessons  learned  were  collected. 
However,  those  activities  were  also  conducted  informally,  without  process 
improvement  necessarily  in  mind. 

The  Case  1  project  team  noted  negative  impacts  to  the  fact  that  the  initial 
set  of  identified  risks  were  not  effectively  tracked,  individual  project  team 
members  did  not  apply  their  own  experience  to  help  in  identifying,  assessing, 
prioritizing,  mitigating,  and  tracking  risks,  project  risks  were  not  presented  as 
effectively  as  they  could  have  been.  The  team  also  recognized  that  more 
complicated  problems  were  being  resolved  but  the  effort  audits  didn’t  show  how 
much  effort  was  expended  on  each  problem.  In  addition,  there  was  a  significant 
lack  of  compliance  in  recording  effort  data,  and  some  metrics,  such  as  effort 
data,  was  difficult  to  collect  manually  and  presented  problems  in  the  staff  not 
understanding  the  benefit  of  their  use.  In  general,  the  project  team 
acknowledged  that  they  did  not  think  they  were  well  informed  of  senior 
management’s  comments  and  directives,  and  that  the  large  all-encompassing 
schedule  became  too  complex  and  unwieldy.  At  first  glance  this  list  seems  long 
compared  to  the  evaluation  results  reported,  but  the  fact  that  there  was  not 
enough  corrective  action  taken  to  correct  process  execution  supports  the  entire 
list. 
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The  Case  2  project  had  additional  practices  that  received  the  “Not 
Performed  Adequately”  rating  (noted  above).  The  ratings  were  a  direct  result  of 
inadequate  risk  monitoring,  not  maintaining  the  plan  for  the  PMC  process,  not 
providing  adequate  resources  and  assigning  responsibility  and  authority  for 
executing  the  PMC  process,  inadequate  stakeholder  involvement,  and  not 
reviewing  issues  with  senior  management  or  using  them  to  help  solve  problems. 

The  Case  2  lessons  learned  was  a  long  list  including  the  fact  that 
schedules  and  key  project  artifacts  were  not  kept  up  to  date  and  visible,  elements 
of  the  plan  were  not  done,  project  metrics  not  accurate  and  were  not  being  used, 
an  overall  project  schedule  was  non-existent,  lack  of  project  manager,  and  a  lack 
of  commitment  to  effort  recording/reporting  (much  missing  data  and  inaccurate 
data,  and  inconsistent  worksheet  organization).  These  are  the  symptoms  noticed 
by  the  project  team  as  a  result  of  confusion  over  the  project  plan,  and  lack  of 
direction  from  a  dedicated  project  engineer  (manager). 

Again,  it  is  interesting  to  note  that  one  characteristic  of  a  CMMI  Level  1 
organization  is  that  people  create  their  own  processes  to  do  the  work,  creating 
the  potential  for  duplication,  and  even  conflict  later  on  in  the  project.  The 
migration  in  effort  recording  activities  demonstrated  this  characteristic. 

While  the  organization  did  not  have  outstanding  performance  on  all 
practices,  it  was  demonstrated  that  something  was  being  done  by  the  project 
team  in  every  area.  The  transition  from  simply  managing  requirements  to 
actually  managing  plans  takes  place  as  the  organization  advances  from  Level  2 
to  Level  3.  It  appears  that  the  source  of  risk  lied  with  maintaining  a  dedicated 
resource  for  a  project  engineer  and  keeping  senior  management  involved  to  help 
enforce  the  discipline  needed  to  make  sure  corrective  action  was  identified  and 
followed  through  with. 

This  information  can  be  used  to  identify  a  business  goal  on  which  to  base 
targeted  process  improvement,  such  as  “to  improve  the  project  monitoring  and 
control  process.” 


79 


E.  SUGGESTED  IMPROVEMENTS 

As  a  result  of  the  above  Case  1  and  Case  2  analysis  the  following  list  of 
improvement  suggestions  was  compiled.  Some  suggestions  are  a  direct 
reflection  of  the  lessons  learned  reported,  some  are  specific  to  CMMI  practice 
requirements  that  need  to  be  fulfilled,  and  some  are  more  general  and  would 
contribute  indirectly  to  all  reported  deficiencies.  It  can  be  used  to  improve  the 
project  performance  by  targeting  process  improvement  in  these  areas. 

•  Make  sure  a  dedicated  resource  is  assigned  to  perform  the  project 
engineer  (manager)  role  (to  follow  up  on  project  plan  peer  review  and 
approval  process,  project  and  senior  management  review  meetings, 
risk  management,  the  change  control  process,  etc.). 

•  Schedules,  plans,  scope  documents,  and  other  and  key  project 
artifacts  need  to  be  kept  up  to  date  and  visible,  presented  at  every 
meeting.  Responsibilities  for  maintenance  of  these  products  must  be 
clearly  assigned.  Use  “baselining”  (accomplished  at  the  organization 
by  completing  the  senior  management  approval  process)  to  keep  a 
record  of  current  agreements  made  for  the  duration  of  the  project. 

•  Training  in  the  risk  management  process  should  be  provided  to  ensure 
that  true  risks  to  project  completion  are  identified,  assessed,  prioritized, 
mitigated,  and  that  contingency  plans  are  in  place  should  a  risk  be 
realized.  The  current  set  of  risks  should  be  available  to  all  project  staff 
in  a  shared  repository.  A  standard  format  for  presenting  the  highest 
priority  risks  at  Senior  Management  Reviews  should  be  developed. 

•  There  should  be  more  detail  recorded  in  planning  and  estimating 
resource  requirements.  The  project  should  be  tracked  at  two  levels  - 
project  and  functional.  All  of  the  schedules  should  be  posted  on  a 
shared  repository  so  that  all  project  staff  are  aware  of  key  dates. 

•  Select  project  metrics  for  the  information  they  provide  and  ensure  their 
ease  of  collection.  More  feedback  on  definition,  collection,  and  use  of 
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metrics  should  be  provided  to  project  team  members.  The  categories, 
method,  frequency  of  collecting  these  records,  and  the  reason  for 
collecting  hours  and  how  this  information  will  be  used  also  needs  to  be 
well  defined  and  fully  explained.  Similar  efforts  are  needed  for  the 
remaining  project  metrics  in  order  to  improve  compliance  by  the  project 
staff. 

•  Provide  more  feedback  to  the  project  staff  on  the  results  of  the  Senior 
Management  Reviews  (SMRs). 

•  Review  and  renew  the  commitment  to  record  and  collect  effort  data  to 
demonstrate  that  this  is  a  valuable  and  necessary  activity.  Develop 
standard  naming  conventions  for  effort  worksheet  file  names  and  effort 
categories,  a  way  to  indicate  when  the  worksheet  was  last  updated, 
and  a  standard  way  to  add/delete  team  members  from  the  audit 
roster. 

•  Develop  questions  and  metrics  to  measure  accomplishments  in 
response  to  the  business  goal  to  “improve  the  project  monitoring  and 
control  process.” 

•  Archive  all  project  planning  work  products  using  the  organization’s 
designated  CM  tool  (ClearCase).  SCM  needs  to  become  a  continuous 
process  involving  participation  by  the  entire  project  staff  (rather  than 
just  a  repository  for  work  products).  Staff  members  should  generate 
Process  Change  Requests  (PCRs)  to  resolve  issues. 

•  Conduct  a  formal  audit  on  the  project  monitoring  and  control  process  at 
least  once  during  project  execution. 

•  Update  policy,  process,  and  procedure  documentation  to  reflect 
additional  activities. 
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VII.  CONCLUSIONS 


This  study  showed  that  while  the  organization  is  performing  to  some 
extent  in  every  area  of  the  project  management  practices  evaluated,  there  is  still 
much  room  for  improvement.  The  evaluation  results  did  not  demonstrate 
improvement  of  any  one  practice,  there  were  many  practices  for  which  both 
projects  were  rated  “Not  Performed  Adequately,”  and  there  were  many  cases  for 
which  the  more  recent  project  (Case  2)  got  a  lower  rating  than  the  less  recent 
project  (Casel),  i.e. ,  evaluation  results  showed  some  digression  from  one  project 
to  the  next.  This  conclusion  was  reached  by  consensus  of  a  small  group  of 
project  team  members  who  participated  in  reviewing  the  results.  It  matched  the 
general  perception  held  by  the  team,  and  management. 

Given  that  the  project  team  was  highly  successful  at  delivering  planned 
scope,  within  planned  schedule  and  fixed  cost,  these  results  are  surprising  since 
team  members  have  remarked  about  how  much  more  organized  and  consistent 
the  process  seems  to  be  than  ever  before  (albeit  they  would  be  referring  to  all 
processes  used,  not  just  project  management  processes).  Why  then  doesn’t 
this  show  up  in  the  evaluation  results? 

The  project  team  is  small  (less  than  20  people),  and  most  of  the  same 
people  fulfilled  the  same  duties  on  both  projects  studied.  The  projects  were 
similar  (unique  iterations  of  software  maintenance  efforts),  and  the  staff  were 
experienced  (from  previous  iterations  of  the  same  type  of  projects).  The 
advantage  of  the  circumstances  implies  that  the  team  was  able  to  rely  on 
experienced  staff  to  do  whatever  needed  to  be  done  to  make  the  delivery  on 
time,  in  spite  of  process  digression. 

So  why  should  the  organization  even  do  process  improvement  since 
achieving  a  certain  CMMI  level  cannot  improve  on  the  already  great  record? 
One  valid  reason  might  be  to  satisfy  a  bidding  requirement  to  gain  more 
government  contracts,  to  in  effect  “keep  up  with  industry  trends”  by  advertising 
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the  organization’s  achievement.  But  what  is  the  internal  value  for  this  pursuit? 
Does  the  organization  need  to  improve  its  effectiveness  and  efficiency? 

One  way  to  answer  this  question  is  to  look  closer  at  one  aspect  of  this 
evaluation  -recording,  collection,  analysis,  and  reporting  of  effort  data.  The 
project  team  may  ask  themselves  if  they  even  know  how  effectively  and 
efficiently  they  perform  various  tasks?  There  were  many  problems  related  to 
these  activities  (determining  estimates  versus  actual  time  spent  on  categories  of 
tasks).  Improving  this  process  would  help  the  team  to  gather  more  accurate 
data,  which  would  enable  the  team  to  discover  what  the  measurements  mean  in 
terms  of  productivity.  Once  understood  quantitatively,  process  improvements 
could  be  targeted  for  certain  processes  that  affect  effectiveness  and  efficiency. 
More  important,  the  organization  will  have  the  capability  to  demonstrate  these 
skills  to  customers  with  data,  instead  of  opinions.  This  could  be  used  as  a 
valuable  marketing  tool. 

Further,  as  processes  become  more  consistent,  individual  frustrations  with 
ad  hoc  methods  will  begin  to  disappear.  When  the  team  begins  to  demonstrate 
enough  discipline  to  learn,  perform,  and  improve  the  same  processes,  the  team 
will  come  together  to  do  more  of  the  same  work  (instead  of  each  person 
improving  their  own  way  of  doing  things,  which  is  likely  different  that  the  way 
another  person  performs  the  same  task). 

One  test  of  process  improvement  is  whether  or  not  it  can  withstand 
changes  in  business  and  personnel.  The  right  combination  of  striving  for 
consistency  and  then  accepting  more  change  will  provide  a  successful  balance. 
The  organization  just  needs  to  know  where  to  start. 

It  sometimes  seems  that  examination  and  acknowledgement  of  the 
reasons  previous  process  improvement  efforts  did  not  take  hold  is  too  often 
glossed  over,  and  the  information  not  used.  Getting  comfortable  with  exposing 
barriers  could  be  the  very  key  to  making  a  change  stick,  or  at  least  help  to 
prevent  doing  the  same  thing,  and  getting  the  same  (not  as  good  as  we  want) 
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results.  Even  though  the  barriers  do  not  usually  turn  out  to  be  rocket  science,  it 
is  often  difficult  to  uncover  and  overcome  them,  especially  in  an  organization 
where  things  have  been  done  in  the  same  ways,  and  by  the  same  people,  for 
years.  Add  that  to  the  fact  that  making  even  minor  changes  for  most  of  us  is  at 
best  challenging.  “Starting  over"  has  to  be  a  way  of  life  to  get  different  results. 
And  there  has  to  be  some  strong  motivation,  even  for  simple  changes.  If  the 
organization  is  committed  to  preparation  for  expanding  the  ability  to  grow  the 
SOW  it  must  be  strongly  emphasized. 
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VIII.  POTENTIAL  BENEFITS 


The  following  list  outlines  some  predicted  benefits  that  should  take  place 
after  implementation  of  the  suggested  process  improvements  discussed  in  this 
paper.  Some  benefits  of  this  study  have  already  been  realized,  as  all  of  the  work 
done  by  this  organization  in  the  last  few  years  have  planted  seeds  that  are  now 
presenting  opportunities.  Conducting  this  study  has  served  as  a  training  tool 
among  the  participants  and  stirred  many  very  good  additional  questions. 

A.  INTERNAL  TO  THE  ORGANIZATION 

The  primary  benefits  internal  to  the  organization  include: 

•  Increased  Awareness  of  the  current  organization  processes,  how  they 
compare  to  CMMI  practices,  and  specific  ideas  for  valuable 
improvements 

•  A  better  understanding  of  the  value  of  collecting  and  using  lessons 
learned 

•  An  improved  problem  solving  environment  (better  communication 
leads  to  greater  understanding,  which  translates  into 
acknowledgement,  then  comfort) 

•  Opening  and  strengthening  communication  between  the  project  team 
and  senior  management  will  help  to  uncover  and  prevent  further 
misunderstanding  about  current  processes  as  they  work  together  to 
devise  the  best  process  solutions 

•  Improvement  suggestions  that  are  related  to  low  level  business  goals 
(i.e.,  “improve  the  requirements  management  process”  is  easier  to 
measure  than  to  “improve  productivity”)  can  be  prioritized  according  to 
management  needs 

•  The  organization  will  learn  to  plan  to  prevent  problems,  instead  of  just 
reacting  to  problems. 

•  Improved  basic  project  management  skills  (planning,  tracking,  and 
reporting) 
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•  Improved  mid-course  decision  making  to  help  manage  risks 

•  Increased  ability  to  provide  more  accurate  estimate  and  progress 
information  to  the  project  team,  management,  industry  partners,  and 
customers. 

•  The  team  will  become  more  enthusiastic  as  success  is  realized  by 
owning  the  culture  shift  from  relying  on  previous  experience  to 
currently  defined  process  discipline. 

•  The  data  needed  to  do  earned  value  progress  reporting  will  be 
available  (a  method  of  relating  the  scope  of  work  to  the  project 
schedule  and  budget). 

•  Increased  productivity  when  goals  between  project  teams  and  senior 
management  are  realigned  and  decisions  made  to  enforce 
organization  level  consistency  in  implementation,  so  the  team  would  be 
able  to  finish  the  project  sooner,  or  take  on  more  scope. 

•  CMMI  achievement  award  of  as  a  result  of  a  SCAMPI  or  official 
licensed  CMMI  appraisal. 

B.  EXTERNAL  TO  THE  ORGANIZATION 

Aside  from  some  of  the  same  benefits  noted  above  being  noticed  by 
stakeholders  who  interact  with  the  organization,  some  additional  benefits  that 
may  be  realized  include  the  following. 

•  Demonstrated  contribution  to  the  ARMY  process  improvement  initiative 

•  A  new  credibility  that  is  recognized  in  the  industry,  and  can  be  used  as 
a  strong  marketing  tool  for  the  organization,  and  for  the  Army 


88 


LIST  OF  REFERENCES 


1.  Mark  Paulk,  Bill  Curtis,  Mary  Beth  Chrissis,  Charles  Weber,  “Capability 
Maturity  Model  for  Software,  Version  1.1”,  CMU/SEI-93-TR-24  (1993). 

2.  CMMI  Product  Team,  “Capability  Maturity  Model®  Integration  (CMMIsm), 
Version  1.1,  CMMIsm  for  Systems  Engineering,  Software  Engineering, 
Integrated  Product  and  Process  Development,  and  Supplier  Sourcing 
(CMMI-SE/SW/IPPD/SS,  VI. 1),  Staged  Representation”,  CMU/SEI-2002- 
TR-012  (2002). 

3.  Watts  S.  Humphrey,  “Managing  the  Software  Process”,  Software 
Engineering  Institute,  1990. 

4.  Mary  Beth  Chrissis,  Mike  Konrad,  Sandy  Shrum,  “CMMI,  Guidelines  for 
Process  Integration  and  Product  Improvement”,  Software  Engineering 
Institute  Series  in  Software  Engineering,  2003. 

5.  Office  of  the  Secretary  of  Defense,  Edward  Aldridge,  Under  Secretary  of 
Defense  (Acquisition,  Technology  and  Logistics),  “Memorandum  for 
Secretaries  of  the  Military  Departments”  with  reference  to  Section  804  of 
the  Bob  Stump  National  Defense  Authorization  Act  for  Fiscal  Year  2003 
which  requires  establishment  of  software  acquisition  process  improvement 
programs  by  Military  Departments  and  those  Defense  Agencies  that 
manage  Major  Defense  Acquisition  Programs  with  a  substantial 
component),  March  2003. 

6.  United  States  General  Accountability  Office,  “Report  to  the  Committee  on 
Armed  Services,  U.  S.  Senate,  Defense  Acquisitions,  Stronger 
Management  Practices  Are  Needed  to  Improve  DOD’s 
Software-Intensive  Weapon  Acquisitions,”  March  2004. 

7.  Research,  Development,  and  Acquisition,  Army  Acquisition  Procedures, 
Department  of  the  Army  Pamphlet  70-3,  Headquarters,  Department  of  the 
Army,  Washington,  DC,  15  July  1999. 

8.  Defense  Acquisition  University,  ACQ  -  201 B  Intermediate  Systems 
Acquisition  Course,  Student  Book,  IPT  Workshop  CD,  December  2002. 

9.  Under  Secretary  of  the  Air  Force,  Memorandum  -  “Revitalizing  the 
Software  Aspects  of  Systems  Engineering”,  2  September  2004. 


89 


10. “Interpreting  the  CMMI,  A  Process  Improvement  Approach”,  Margaret  K. 
Kulpa,  Kent  A.  Johnson,  2003,  Appendix  B,  “Myths  and  Legends  of  the 
CMMI”,  page  293,  “CMMI  supports  organizational  business  objectives.” 


90 


APPENDIX  -  SAMPLE  PROCESS  AREA  EVALUATION  CHECKLIST 


Evaluator:  SQA  Engineer 
Assessment  Date:  1/24/06  -  3/10/06 

Project  Audited:  Case  1  and  Case  2  -  Evaluating  both  projects  was 
recommended  by  the  SEPG  since  it  will  demonstrate  continuous  use  of 
processes  and  artifacts.  In  addition,  it  will  expose  improvements,  or  regressions, 
from  capabilities  demonstrated  during  the  project  to  capabilities  demonstrated 
during  the  Case  2  project. 

Support/Review  Team:  Project  manager,  Software  Development  Lead,  Test 
Lead,  SEPG  Lead,  SQA  Engineer,  Project  Manager,  and  CM  Lead. 


Specific  Practice  (SP) 

Requirements  Management  -  SP 

[Comments  regarding  interpretation 

Rating 

of  practices  as  applicable  to  this 

NP  -  Not  Performed,  NPA  -  Not 

project  were  inserted  as  needed.] 

Performed  Adequately, 

PA- 

-  Performed  Adequately  or  Better 

NP 

NPA 

PA 

Basis  for 

Rating 

Specific  Goal  (SG)  1  Requirements  are  managed  and  inconsistencies 

with  project  plans  and  work  products  are  identified. 

SP1.1  Develop  an  understanding  with 
the  requirements  providers  on  the 
meaning  of  the  requirements  [budget, 
schedule,  and  delivery  constraints]. 

Case  1 

Case  2 

***  Note  that 
all  comments 
in  this  column 
were  removed 
for 

presentation 
of  results  in 
this  paper  to 
maintain 
project 
anonymity. 
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SP1 .2  Obtain  Commitment  to  the 
requirements  from  the  project 
participants.  This  practice  refers  to 
the  “initial”  commitment  (not  changes) 
to  a  specific  set  of  requirements. 

Case 

2 

Case  1 

SP1 .3  Manage  changes  to  the 
requirements  as  they  evolve  during 
the  project. 

This  practice  refers  only  to  changes 
made  to  the  initial  baseline  to  a 
specific  set  of  requirements. 

Case 

1 

Case 

2 

SP1.4  Maintain  bidirectional 
traceability  among  the  requirements 
and  the  project  plans  and  the  work 
products.  This  is  interpreted  as 
referring  to  both  the  initial  and 
changes  made  to  the  requirements. 

For  the  purposes  of  this  study,  it  was 
interpreted  as  traceability  between 
the  external  Statement  of  Work 
(SOW)  to  the  organizations  internal 
SOW,  and  to  the  organizations  Case 

1  and  Case  2  project  plans.  In  both 
cases,  the  organization’s  internal 
schedule  was  considered  part  of  the 
plans. 

Case 

1 

Case 

2 

SP1 .5  Identify  inconsistencies 
between  the  project  plans  and  work 
products  and  the  requirements.  This 
is  interpreted  as  comparison  between 
the  (project  plans  and/or  the  work 
products)  and  the  requirements. 

Case 

2 

Case  1 

Generic  Practice  (GP) 

Requirements  Management  -  GP  Rating 

NP 

NPA 

PA 

Basis  for  Rating 

Note  that  the 

comments  in  this 

column  were 

removed  for 

92 


presentation  of 

results  in  this 

paper  to  maintain 

project 

anonymity. 

Generic  Goal  (GG)  2  -  The  process  is  institutionalized  as  a  managed  process. 

GP2.1  -  Establish  and  maintain  an 
organizational  policy  for  planning 
and  performing  the  Requirements 
Management  process. 

Case  1 

Case  2 

GP2.2  -  Establish  and  maintain 
the  plan  for  performing  the 
Requirements  Management 
process. 

Case  1 

Case  2 

GP2.3  -  Provide  adequate 
resources  for  performing  the 
Requirements  Management 
process,  developing  the  work 
products,  and  providing  the 
services  of  the  process. 

Case  1 

Case  2 

GP2.4  -  Assign  responsibility  and 
authority  for  performing  the 
process,  developing  the  work 
products,  and  providing  the 
services  of  the  Requirements 
Management  process. 

Case  1 

Case  2 

GP2.5  -  Train  the  people 
performing  or  supporting  the 
Requirements  Management 
process  as  needed. 

Case  1 

Case  2 

GP2.6  -  Place  designated  work 
products  of  the  Requirements 
Management  process  under 
appropriate  levels  of  configuration. 

Case 

1 

Case 

2 

GP  2.7  -  Identify  and  involve  the 
relevant  stakeholders  of  the 
Requirements  Management 
process  as  planned. 

Case  1 

Case  2 
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GP2.8  -  Monitor  and  control  the 
Requirements  Management 

Case 

Case  1 

process  against  the  plan  for 
performing  the  process  and  take 
appropriate  corrective  action. 

2 

GP2.9  -  Objectively  evaluate 
adherence  of  the  Requirements 
Management  process  against  its 

Case 

1 

process  description,  standards, 

Case 

and  procedures,  and  address 
noncompliance. 

2 

GP2.10  -  Review  the  activities, 
status,  and  results  of  the 

Case 

Case  1 

Requirements  Management 
process  with  higher  level 
management  and  resolve  issues. 

2 

GG3  -  The  process  is  institutionalized  a  defined  process. 

GP  3.1  Establish  and  maintain  the 

Case  1 

description  of  a  defined 
Requirements  Management 
process. 

Case  2 

GP  3.2  Collect  work  products, 

Case 

measures,  measurement  results, 

1 

and  improvement  information 

Case 

derived  from  planning  and 
performing  the  Requirements 
Management  process  to  support 
the  future  use  and  improvement  of 
the  organization’s  processes  and 
process  assets. 

2 

94 


INITIAL  DISTRIBUTION  LIST 


1.  Defense  Technical  Information  Center 
Ft.  Belvoir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 

3.  Dr.  Man-Tak  Shing 

Naval  Postgraduate  School 
Monterey,  California 

4.  Dr.  Peter  Denning 

Naval  Postgraduate  School 
Monterey,  California 

5.  Magid  Athnasios 

United  States  Army  RDECOM 
AMSRD-TAR-R  /  MS-265 
Warren,  Michigan 

6.  Mark  Slominski 

United  States  Army  RDECOM 
AMSRD-TAR-R  /  MS-265 
Warren,  Michigan 

7.  Edward  Andres 

United  States  Army  RDECOM 
AMSRD-TAR-R  /  MS-265 
Warren,  Michigan 

8.  Russell  Menko 

United  States  Army  RDECOM 
AMSRD-TAR-R  /  MS-265 
Warren,  Michigan 

9.  Jack  Platt 

United  States  Army  RDECOM 
AMSRD-TAR-R  /  MS-265 
Warren,  Michigan 


95 


10.  Joe  Szafranski 

United  States  Army  RDECOM 
AMSRD-TAR-R  /  MS-265 
Warren,  Michigan 

1 1 .  Karen  LaFond 

United  States  Army  RDECOM 
AMSRD-TAR-R  /  MS-265 
Warren,  Michigan 


96 


